Sep 14 09:43:29.685088 ip-10-0-132-4 systemd[1]: Starting Kubernetes Kubelet...
Sep 14 09:43:30.078440 ip-10-0-132-4 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 14 09:43:30.078440 ip-10-0-132-4 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 14 09:43:30.078440 ip-10-0-132-4 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 14 09:43:30.078440 ip-10-0-132-4 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 14 09:43:30.078440 ip-10-0-132-4 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 14 09:43:30.079992 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.079883    2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 14 09:43:30.083662 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083632    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 14 09:43:30.083662 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083660    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 14 09:43:30.083662 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083665    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 14 09:43:30.083778 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083669    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 14 09:43:30.083778 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083672    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 14 09:43:30.083778 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083675    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 14 09:43:30.083778 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083678    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 14 09:43:30.083778 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083683    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 14 09:43:30.083778 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083700    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 14 09:43:30.083778 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083705    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 14 09:43:30.083778 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083709    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 14 09:43:30.083778 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083713    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 14 09:43:30.083778 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083716    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 14 09:43:30.083778 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083721    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 14 09:43:30.083778 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083725    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 14 09:43:30.083778 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083729    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 14 09:43:30.083778 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083732    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 14 09:43:30.083778 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083735    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 14 09:43:30.083778 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083738    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 14 09:43:30.083778 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083741    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 14 09:43:30.083778 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083751    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 14 09:43:30.083778 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083756    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 14 09:43:30.083778 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083761    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 14 09:43:30.084271 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083764    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 14 09:43:30.084271 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083767    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 14 09:43:30.084271 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083769    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 14 09:43:30.084271 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083772    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 14 09:43:30.084271 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083775    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 14 09:43:30.084271 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083778    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 14 09:43:30.084271 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083782    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 14 09:43:30.084271 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083785    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 14 09:43:30.084271 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083789    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 14 09:43:30.084271 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083793    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 14 09:43:30.084271 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083796    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 14 09:43:30.084271 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083798    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 14 09:43:30.084271 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083801    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 14 09:43:30.084271 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083804    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 14 09:43:30.084271 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083806    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 14 09:43:30.084271 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083809    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 14 09:43:30.084271 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083811    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 14 09:43:30.084271 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083814    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 14 09:43:30.084271 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083817    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 14 09:43:30.084271 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083819    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 14 09:43:30.084793 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083822    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 14 09:43:30.084793 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083825    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 14 09:43:30.084793 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083827    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 14 09:43:30.084793 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083830    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 14 09:43:30.084793 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083833    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 14 09:43:30.084793 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083835    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 14 09:43:30.084793 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083838    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 14 09:43:30.084793 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083842    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 14 09:43:30.084793 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083845    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 14 09:43:30.084793 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083848    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 14 09:43:30.084793 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083851    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 14 09:43:30.084793 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083854    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 14 09:43:30.084793 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083856    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 14 09:43:30.084793 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083860    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 14 09:43:30.084793 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083863    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 14 09:43:30.084793 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083865    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 14 09:43:30.084793 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083868    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 14 09:43:30.084793 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083871    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 14 09:43:30.084793 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083873    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 14 09:43:30.085295 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083876    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 14 09:43:30.085295 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083878    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 14 09:43:30.085295 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083881    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 14 09:43:30.085295 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083884    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 14 09:43:30.085295 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083886    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 14 09:43:30.085295 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083889    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 14 09:43:30.085295 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083892    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 14 09:43:30.085295 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083894    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 14 09:43:30.085295 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083897    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 14 09:43:30.085295 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083900    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 14 09:43:30.085295 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083903    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 14 09:43:30.085295 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083905    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 14 09:43:30.085295 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083908    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 14 09:43:30.085295 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083910    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 14 09:43:30.085295 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083913    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 14 09:43:30.085295 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083917    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 14 09:43:30.085295 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083921    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 14 09:43:30.085295 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083924    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 14 09:43:30.085295 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083927    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 14 09:43:30.085295 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083929    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 14 09:43:30.085812 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083932    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 14 09:43:30.085812 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083934    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 14 09:43:30.085812 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083937    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 14 09:43:30.085812 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083939    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 14 09:43:30.085812 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083942    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 14 09:43:30.085812 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083944    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 14 09:43:30.085812 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083948    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 14 09:43:30.085812 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083951    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 14 09:43:30.085812 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083954    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 14 09:43:30.085812 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083956    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 14 09:43:30.085812 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083960    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 14 09:43:30.085812 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083963    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 14 09:43:30.085812 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083966    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 14 09:43:30.085812 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083969    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 14 09:43:30.085812 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083972    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 14 09:43:30.085812 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083974    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 14 09:43:30.085812 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.083977    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 14 09:43:30.085812 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084746    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 14 09:43:30.085812 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084755    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 14 09:43:30.086309 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084759    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 14 09:43:30.086309 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084763    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 14 09:43:30.086309 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084765    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 14 09:43:30.086309 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084769    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 14 09:43:30.086309 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084772    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 14 09:43:30.086309 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084775    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 14 09:43:30.086309 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084778    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 14 09:43:30.086309 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084781    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 14 09:43:30.086309 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084784    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 14 09:43:30.086309 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084787    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 14 09:43:30.086309 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084790    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 14 09:43:30.086309 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084793    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 14 09:43:30.086309 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084795    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 14 09:43:30.086309 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084798    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 14 09:43:30.086309 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084801    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 14 09:43:30.086309 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084803    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 14 09:43:30.086309 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084806    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 14 09:43:30.086309 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084809    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 14 09:43:30.086309 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084811    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 14 09:43:30.086309 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084814    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 14 09:43:30.086832 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084817    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 14 09:43:30.086832 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084820    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 14 09:43:30.086832 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084823    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 14 09:43:30.086832 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084825    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 14 09:43:30.086832 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084828    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 14 09:43:30.086832 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084830    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 14 09:43:30.086832 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084834    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 14 09:43:30.086832 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084837    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 14 09:43:30.086832 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084839    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 14 09:43:30.086832 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084842    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 14 09:43:30.086832 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084846    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 14 09:43:30.086832 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084848    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 14 09:43:30.086832 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084851    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 14 09:43:30.086832 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084855    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 14 09:43:30.086832 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084859    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 14 09:43:30.086832 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084862    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 14 09:43:30.086832 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084865    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 14 09:43:30.086832 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084868    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 14 09:43:30.086832 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084872    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 14 09:43:30.087350 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084876    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 14 09:43:30.087350 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084880    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 14 09:43:30.087350 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084883    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 14 09:43:30.087350 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084886    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 14 09:43:30.087350 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084889    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 14 09:43:30.087350 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084892    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 14 09:43:30.087350 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084895    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 14 09:43:30.087350 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084898    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 14 09:43:30.087350 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084900    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 14 09:43:30.087350 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084902    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 14 09:43:30.087350 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084905    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 14 09:43:30.087350 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084908    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 14 09:43:30.087350 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084911    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 14 09:43:30.087350 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084913    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 14 09:43:30.087350 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084916    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 14 09:43:30.087350 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084919    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 14 09:43:30.087350 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084921    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 14 09:43:30.087350 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084924    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 14 09:43:30.087350 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084926    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 14 09:43:30.087350 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084929    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 14 09:43:30.087878 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084932    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 14 09:43:30.087878 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084935    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 14 09:43:30.087878 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084937    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 14 09:43:30.087878 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084940    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 14 09:43:30.087878 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084943    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 14 09:43:30.087878 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084945    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 14 09:43:30.087878 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084948    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 14 09:43:30.087878 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084951    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 14 09:43:30.087878 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084953    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 14 09:43:30.087878 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084956    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 14 09:43:30.087878 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084965    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 14 09:43:30.087878 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084968    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 14 09:43:30.087878 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084971    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 14 09:43:30.087878 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084973    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 14 09:43:30.087878 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084976    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 14 09:43:30.087878 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084979    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 14 09:43:30.087878 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084981    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 14 09:43:30.087878 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084984    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 14 09:43:30.087878 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084987    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 14 09:43:30.087878 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084990    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 14 09:43:30.088373 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084992    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 14 09:43:30.088373 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084995    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 14 09:43:30.088373 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.084998    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 14 09:43:30.088373 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085000    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 14 09:43:30.088373 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085003    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 14 09:43:30.088373 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085006    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 14 09:43:30.088373 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085009    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 14 09:43:30.088373 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085011    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 14 09:43:30.088373 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085014    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 14 09:43:30.088373 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085017    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 14 09:43:30.088373 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085019    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 14 09:43:30.088373 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085022    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 14 09:43:30.088373 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085025    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 14 09:43:30.088373 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085028    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 14 09:43:30.088373 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085030    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 14 09:43:30.088373 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085033    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 14 09:43:30.088373 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085035    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 14 09:43:30.088373 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085039    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 14 09:43:30.088373 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085464    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 14 09:43:30.088373 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085469    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 14 09:43:30.088915 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085472    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 14 09:43:30.088915 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085475    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 14 09:43:30.088915 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085479    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 14 09:43:30.088915 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085482    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 14 09:43:30.088915 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085485    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 14 09:43:30.088915 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085488    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 14 09:43:30.088915 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085491    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 14 09:43:30.088915 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085493    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 14 09:43:30.088915 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085496    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 14 09:43:30.088915 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085499    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 14 09:43:30.088915 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085502    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 14 09:43:30.088915 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085504    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 14 09:43:30.088915 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085507    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 14 09:43:30.088915 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085509    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 14 09:43:30.088915 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085512    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 14 09:43:30.088915 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085515    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 14 09:43:30.088915 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085518    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 14 09:43:30.088915 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085521    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 14 09:43:30.088915 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085524    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 14 09:43:30.088915 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085526    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 14 09:43:30.089417 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085531    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 14 09:43:30.089417 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085534    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 14 09:43:30.089417 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085538    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 14 09:43:30.089417 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085540    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 14 09:43:30.089417 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085543    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 14 09:43:30.089417 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085546    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 14 09:43:30.089417 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085549    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 14 09:43:30.089417 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085552    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 14 09:43:30.089417 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085554    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 14 09:43:30.089417 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085557    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 14 09:43:30.089417 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085560    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 14 09:43:30.089417 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085562    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 14 09:43:30.089417 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085565    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 14 09:43:30.089417 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085568    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 14 09:43:30.089417 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085571    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 14 09:43:30.089417 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085574    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 14 09:43:30.089417 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085578    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 14 09:43:30.089417 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085585    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 14 09:43:30.089417 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085588    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 14 09:43:30.089918 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085591    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 14 09:43:30.089918 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085594    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 14 09:43:30.089918 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085597    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 14 09:43:30.089918 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085600    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 14 09:43:30.089918 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085602    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 14 09:43:30.089918 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085605    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 14 09:43:30.089918 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085608    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 14 09:43:30.089918 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085610    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 14 09:43:30.089918 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085613    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 14 09:43:30.089918 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085616    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 14 09:43:30.089918 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085618    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 14 09:43:30.089918 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085622    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 14 09:43:30.089918 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085624    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 14 09:43:30.089918 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085627    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 14 09:43:30.089918 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085630    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 14 09:43:30.089918 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085633    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 14 09:43:30.089918 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085635    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 14 09:43:30.089918 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085638    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 14 09:43:30.089918 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085641    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 14 09:43:30.089918 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085643    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 14 09:43:30.090414 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085646    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 14 09:43:30.090414 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085648    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 14 09:43:30.090414 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085651    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 14 09:43:30.090414 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085653    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 14 09:43:30.090414 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085657    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 14 09:43:30.090414 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085659    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 14 09:43:30.090414 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085662    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 14 09:43:30.090414 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085664    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 14 09:43:30.090414 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085667    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 14 09:43:30.090414 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085671    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 14 09:43:30.090414 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085676    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 14 09:43:30.090414 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085679    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 14 09:43:30.090414 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085681    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 14 09:43:30.090414 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085684    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 14 09:43:30.090414 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085703    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 14 09:43:30.090414 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085707    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 14 09:43:30.090414 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085711    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 14 09:43:30.090414 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085715    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 14 09:43:30.090414 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085720    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 14 09:43:30.090891 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085725    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 14 09:43:30.090891 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085728    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 14 09:43:30.090891 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085731    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 14 09:43:30.090891 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085734    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 14 09:43:30.090891 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085736    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 14 09:43:30.090891 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085739    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 14 09:43:30.090891 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085742    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 14 09:43:30.090891 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085744    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 14 09:43:30.090891 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085747    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 14 09:43:30.090891 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085750    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 14 09:43:30.090891 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085752    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 14 09:43:30.090891 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085755    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 14 09:43:30.090891 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085757    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 14 09:43:30.090891 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085760    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 14 09:43:30.090891 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085762    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 14 09:43:30.090891 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085765    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 14 09:43:30.090891 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085768    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 14 09:43:30.090891 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085770    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 14 09:43:30.090891 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.085773    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 14 09:43:30.090891 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.085873    2563 flags.go:64] FLAG: --address="0.0.0.0"
Sep 14 09:43:30.091386 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.085926    2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 14 09:43:30.091386 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.085933    2563 flags.go:64] FLAG: --anonymous-auth="true"
Sep 14 09:43:30.091386 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.085938    2563 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 14 09:43:30.091386 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.085946    2563 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 14 09:43:30.091386 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.085950    2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 14 09:43:30.091386 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.085954    2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 14 09:43:30.091386 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.085959    2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 14 09:43:30.091386 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.085962    2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 14 09:43:30.091386 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.085965    2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 14 09:43:30.091386 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.085969    2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 14 09:43:30.091386 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.085972    2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 14 09:43:30.091386 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.085976    2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 14 09:43:30.091386 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.085979    2563 flags.go:64] FLAG: --cgroup-root=""
Sep 14 09:43:30.091386 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.085982    2563 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 14 09:43:30.091386 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.085985    2563 flags.go:64] FLAG: --client-ca-file=""
Sep 14 09:43:30.091386 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.085988    2563 flags.go:64] FLAG: --cloud-provider="external"
Sep 14 09:43:30.091386 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.085991    2563 flags.go:64] FLAG: --cluster-dns="[]"
Sep 14 09:43:30.091386 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.085996    2563 flags.go:64] FLAG: --cluster-domain=""
Sep 14 09:43:30.091386 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.085999    2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 14 09:43:30.091386 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086002    2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 14 09:43:30.091386 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086006    2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 14 09:43:30.091386 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086009    2563 flags.go:64] FLAG: --container-log-max-files="5"
Sep 14 09:43:30.091386 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086013    2563 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 14 09:43:30.091386 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086017    2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 14 09:43:30.091994 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086020    2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 14 09:43:30.091994 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086023    2563 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 14 09:43:30.091994 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086026    2563 flags.go:64] FLAG: --contention-profiling="false"
Sep 14 09:43:30.091994 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086029    2563 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 14 09:43:30.091994 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086032    2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 14 09:43:30.091994 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086036    2563 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 14 09:43:30.091994 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086039    2563 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 14 09:43:30.091994 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086043    2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 14 09:43:30.091994 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086046    2563 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 14 09:43:30.091994 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086049    2563 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 14 09:43:30.091994 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086052    2563 flags.go:64] FLAG: --enable-load-reader="false"
Sep 14 09:43:30.091994 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086057    2563 flags.go:64] FLAG: --enable-server="true"
Sep 14 09:43:30.091994 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086061    2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 14 09:43:30.091994 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086066    2563 flags.go:64] FLAG: --event-burst="100"
Sep 14 09:43:30.091994 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086069    2563 flags.go:64] FLAG: --event-qps="50"
Sep 14 09:43:30.091994 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086072    2563 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 14 09:43:30.091994 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086075    2563 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 14 09:43:30.091994 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086079    2563 flags.go:64] FLAG: --eviction-hard=""
Sep 14 09:43:30.091994 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086085    2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 14 09:43:30.091994 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086088    2563 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 14 09:43:30.091994 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086091    2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 14 09:43:30.091994 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086094    2563 flags.go:64] FLAG: --eviction-soft=""
Sep 14 09:43:30.091994 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086097    2563 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 14 09:43:30.091994 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086101    2563 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 14 09:43:30.091994 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086104    2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 14 09:43:30.092628 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086107    2563 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 14 09:43:30.092628 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086110    2563 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 14 09:43:30.092628 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086113    2563 flags.go:64] FLAG: --fail-swap-on="true"
Sep 14 09:43:30.092628 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086116    2563 flags.go:64] FLAG: --feature-gates=""
Sep 14 09:43:30.092628 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086120    2563 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 14 09:43:30.092628 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086123    2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 14 09:43:30.092628 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086126    2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 14 09:43:30.092628 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086130    2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 14 09:43:30.092628 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086133    2563 flags.go:64] FLAG: --healthz-port="10248"
Sep 14 09:43:30.092628 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086136    2563 flags.go:64] FLAG: --help="false"
Sep 14 09:43:30.092628 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086139    2563 flags.go:64] FLAG: --hostname-override="ip-10-0-132-4.ec2.internal"
Sep 14 09:43:30.092628 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086142    2563 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 14 09:43:30.092628 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086145    2563 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 14 09:43:30.092628 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086148    2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 14 09:43:30.092628 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086152    2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 14 09:43:30.092628 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086155    2563 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 14 09:43:30.092628 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086158    2563 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 14 09:43:30.092628 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086161    2563 flags.go:64] FLAG: --image-service-endpoint=""
Sep 14 09:43:30.092628 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086165    2563 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 14 09:43:30.092628 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086168    2563 flags.go:64] FLAG: --kube-api-burst="100"
Sep 14 09:43:30.092628 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086172    2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 14 09:43:30.092628 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086175    2563 flags.go:64] FLAG: --kube-api-qps="50"
Sep 14 09:43:30.092628 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086178    2563 flags.go:64] FLAG: --kube-reserved=""
Sep 14 09:43:30.092628 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086181    2563 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 14 09:43:30.093497 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086184    2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 14 09:43:30.093497 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086189    2563 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 14 09:43:30.093497 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086192    2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 14 09:43:30.093497 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086195    2563 flags.go:64] FLAG: --lock-file=""
Sep 14 09:43:30.093497 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086198    2563 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 14 09:43:30.093497 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086201    2563 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 14 09:43:30.093497 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086204    2563 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 14 09:43:30.093497 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086210    2563 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 14 09:43:30.093497 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086213    2563 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 14 09:43:30.093497 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086216    2563 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 14 09:43:30.093497 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086219    2563 flags.go:64] FLAG: --logging-format="text"
Sep 14 09:43:30.093497 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086222    2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 14 09:43:30.093497 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086225    2563 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 14 09:43:30.093497 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086228    2563 flags.go:64] FLAG: --manifest-url=""
Sep 14 09:43:30.093497 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086231    2563 flags.go:64] FLAG: --manifest-url-header=""
Sep 14 09:43:30.093497 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086236    2563 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 14 09:43:30.093497 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086239    2563 flags.go:64] FLAG: --max-open-files="1000000"
Sep 14 09:43:30.093497 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086243    2563 flags.go:64] FLAG: --max-pods="110"
Sep 14 09:43:30.093497 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086246    2563 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 14 09:43:30.093497 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086249    2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 14 09:43:30.093497 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086251    2563 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 14 09:43:30.093497 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086255    2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 14 09:43:30.093497 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086258    2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 14 09:43:30.093497 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086261    2563 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 14 09:43:30.093497 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086264    2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 14 09:43:30.094609 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086272    2563 flags.go:64] FLAG: --node-status-max-images="50"
Sep 14 09:43:30.094609 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086277    2563 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 14 09:43:30.094609 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086280    2563 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 14 09:43:30.094609 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086283    2563 flags.go:64] FLAG: --pod-cidr=""
Sep 14 09:43:30.094609 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086291    2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 14 09:43:30.094609 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086298    2563 flags.go:64] FLAG: --pod-manifest-path=""
Sep 14 09:43:30.094609 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086301    2563 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 14 09:43:30.094609 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086304    2563 flags.go:64] FLAG: --pods-per-core="0"
Sep 14 09:43:30.094609 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086307    2563 flags.go:64] FLAG: --port="10250"
Sep 14 09:43:30.094609 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086312    2563 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 14 09:43:30.094609 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086315    2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-054c21f7cef7342fe"
Sep 14 09:43:30.094609 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086318    2563 flags.go:64] FLAG: --qos-reserved=""
Sep 14 09:43:30.094609 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086322    2563 flags.go:64] FLAG: --read-only-port="10255"
Sep 14 09:43:30.094609 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086325    2563 flags.go:64] FLAG: --register-node="true"
Sep 14 09:43:30.094609 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086328    2563 flags.go:64] FLAG: --register-with-taints=""
Sep 14 09:43:30.094609 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086332    2563 flags.go:64] FLAG: --registry-burst="10"
Sep 14 09:43:30.094609 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086335    2563 flags.go:64] FLAG: --registry-qps="5"
Sep 14 09:43:30.094609 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086338    2563 flags.go:64] FLAG: --reserved-cpus=""
Sep 14 09:43:30.094609 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086341    2563 flags.go:64] FLAG: --reserved-memory=""
Sep 14 09:43:30.094609 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086345    2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 14 09:43:30.094609 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086349    2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 14 09:43:30.094609 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086352    2563 flags.go:64] FLAG: --rotate-certificates="false"
Sep 14 09:43:30.094609 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086355    2563 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 14 09:43:30.094609 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086358    2563 flags.go:64] FLAG: --runonce="false"
Sep 14 09:43:30.094609 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086362    2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 14 09:43:30.095631 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086365    2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 14 09:43:30.095631 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086368    2563 flags.go:64] FLAG: --seccomp-default="false"
Sep 14 09:43:30.095631 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086371    2563 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 14 09:43:30.095631 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086374    2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 14 09:43:30.095631 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086377    2563 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 14 09:43:30.095631 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086381    2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 14 09:43:30.095631 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086384    2563 flags.go:64] FLAG: --storage-driver-password="root"
Sep 14 09:43:30.095631 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086387    2563 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 14 09:43:30.095631 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086391    2563 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 14 09:43:30.095631 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086393    2563 flags.go:64] FLAG: --storage-driver-user="root"
Sep 14 09:43:30.095631 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086396    2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 14 09:43:30.095631 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086400    2563 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 14 09:43:30.095631 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086405    2563 flags.go:64] FLAG: --system-cgroups=""
Sep 14 09:43:30.095631 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086408    2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 14 09:43:30.095631 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086414    2563 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 14 09:43:30.095631 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086417    2563 flags.go:64] FLAG: --tls-cert-file=""
Sep 14 09:43:30.095631 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086421    2563 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 14 09:43:30.095631 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086426    2563 flags.go:64] FLAG: --tls-min-version=""
Sep 14 09:43:30.095631 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086428    2563 flags.go:64] FLAG: --tls-private-key-file=""
Sep 14 09:43:30.095631 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086431    2563 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 14 09:43:30.095631 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086435    2563 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 14 09:43:30.095631 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086437    2563 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 14 09:43:30.095631 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086441    2563 flags.go:64] FLAG: --v="2"
Sep 14 09:43:30.095631 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086445    2563 flags.go:64] FLAG: --version="false"
Sep 14 09:43:30.095631 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086449    2563 flags.go:64] FLAG: --vmodule=""
Sep 14 09:43:30.096294 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086453    2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 14 09:43:30.096294 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.086457    2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 14 09:43:30.096294 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086582    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 14 09:43:30.096294 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086586    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 14 09:43:30.096294 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086590    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 14 09:43:30.096294 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086593    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 14 09:43:30.096294 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086596    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 14 09:43:30.096294 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086599    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 14 09:43:30.096294 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086602    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 14 09:43:30.096294 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086605    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 14 09:43:30.096294 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086608    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 14 09:43:30.096294 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086610    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 14 09:43:30.096294 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086613    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 14 09:43:30.096294 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086616    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 14 09:43:30.096294 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086619    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 14 09:43:30.096294 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086621    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 14 09:43:30.096294 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086624    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 14 09:43:30.096294 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086627    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 14 09:43:30.096294 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086630    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 14 09:43:30.096294 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086633    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 14 09:43:30.097203 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086636    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 14 09:43:30.097203 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086638    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 14 09:43:30.097203 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086641    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 14 09:43:30.097203 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086647    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 14 09:43:30.097203 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086650    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 14 09:43:30.097203 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086652    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 14 09:43:30.097203 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086655    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 14 09:43:30.097203 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086658    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 14 09:43:30.097203 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086661    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 14 09:43:30.097203 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086664    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 14 09:43:30.097203 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086666    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 14 09:43:30.097203 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086669    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 14 09:43:30.097203 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086672    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 14 09:43:30.097203 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086675    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 14 09:43:30.097203 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086677    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 14 09:43:30.097203 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086680    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 14 09:43:30.097203 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086683    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 14 09:43:30.097203 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086702    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 14 09:43:30.097203 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086707    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 14 09:43:30.097203 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086711    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 14 09:43:30.098152 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086715    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 14 09:43:30.098152 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086719    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 14 09:43:30.098152 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086722    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 14 09:43:30.098152 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086725    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 14 09:43:30.098152 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086728    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 14 09:43:30.098152 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086731    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 14 09:43:30.098152 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086733    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 14 09:43:30.098152 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086736    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 14 09:43:30.098152 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086739    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 14 09:43:30.098152 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086741    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 14 09:43:30.098152 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086745    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 14 09:43:30.098152 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086749    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 14 09:43:30.098152 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086752    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 14 09:43:30.098152 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086756    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 14 09:43:30.098152 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086760    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 14 09:43:30.098152 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086765    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 14 09:43:30.098152 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086768    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 14 09:43:30.098152 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086770    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 14 09:43:30.098152 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086773    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 14 09:43:30.098152 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086776    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 14 09:43:30.098887 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086778    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 14 09:43:30.098887 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086781    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 14 09:43:30.098887 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086784    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 14 09:43:30.098887 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086786    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 14 09:43:30.098887 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086789    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 14 09:43:30.098887 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086792    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 14 09:43:30.098887 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086794    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 14 09:43:30.098887 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086797    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 14 09:43:30.098887 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086800    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 14 09:43:30.098887 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086802    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 14 09:43:30.098887 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086805    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 14 09:43:30.098887 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086809    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 14 09:43:30.098887 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086813    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 14 09:43:30.098887 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086816    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 14 09:43:30.098887 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086820    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 14 09:43:30.098887 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086823    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 14 09:43:30.098887 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086826    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 14 09:43:30.098887 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086829    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 14 09:43:30.098887 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086832    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 14 09:43:30.099659 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086834    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 14 09:43:30.099659 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086837    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 14 09:43:30.099659 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086840    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 14 09:43:30.099659 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086842    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 14 09:43:30.099659 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086845    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 14 09:43:30.099659 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086848    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 14 09:43:30.099659 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086852    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 14 09:43:30.099659 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086857    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 14 09:43:30.099659 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086861    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 14 09:43:30.099659 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086866    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 14 09:43:30.099659 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086869    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 14 09:43:30.099659 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086872    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 14 09:43:30.099659 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086875    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 14 09:43:30.099659 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086878    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 14 09:43:30.099659 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086881    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 14 09:43:30.099659 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086884    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 14 09:43:30.099659 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086887    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 14 09:43:30.099659 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086890    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 14 09:43:30.099659 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086892    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 14 09:43:30.100358 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086895    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 14 09:43:30.100358 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086898    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 14 09:43:30.100358 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.086901    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 14 09:43:30.100358 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.087358    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 14 09:43:30.100358 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.095413    2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 14 09:43:30.100358 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.095444    2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 14 09:43:30.100358 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.096374    2563 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 14 09:43:30.100358 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.096397    2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 14 09:43:30.100358 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096508    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 14 09:43:30.100358 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096519    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 14 09:43:30.100358 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096524    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 14 09:43:30.100358 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096530    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 14 09:43:30.100358 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096535    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 14 09:43:30.100358 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096539    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 14 09:43:30.100358 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096544    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 14 09:43:30.100877 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096549    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 14 09:43:30.100877 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096554    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 14 09:43:30.100877 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096558    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 14 09:43:30.100877 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096563    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 14 09:43:30.100877 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096567    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 14 09:43:30.100877 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096571    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 14 09:43:30.100877 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096576    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 14 09:43:30.100877 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096580    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 14 09:43:30.100877 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096585    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 14 09:43:30.100877 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096589    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 14 09:43:30.100877 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096593    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 14 09:43:30.100877 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096598    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 14 09:43:30.100877 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096602    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 14 09:43:30.100877 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096606    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 14 09:43:30.100877 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096610    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 14 09:43:30.100877 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096614    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 14 09:43:30.100877 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096619    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 14 09:43:30.100877 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096623    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 14 09:43:30.100877 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096627    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 14 09:43:30.100877 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096631    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 14 09:43:30.101512 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096635    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 14 09:43:30.101512 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096640    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 14 09:43:30.101512 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096647    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 14 09:43:30.101512 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096652    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 14 09:43:30.101512 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096659    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 14 09:43:30.101512 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096665    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 14 09:43:30.101512 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096671    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 14 09:43:30.101512 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096677    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 14 09:43:30.101512 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096681    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 14 09:43:30.101512 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096699    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 14 09:43:30.101512 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096707    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 14 09:43:30.101512 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096721    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 14 09:43:30.101512 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096728    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 14 09:43:30.101512 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096732    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 14 09:43:30.101512 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096736    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 14 09:43:30.101512 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096741    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 14 09:43:30.101512 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096745    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 14 09:43:30.101512 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096750    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 14 09:43:30.101512 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096754    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 14 09:43:30.102007 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096758    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 14 09:43:30.102007 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096766    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 14 09:43:30.102007 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096770    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 14 09:43:30.102007 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096777    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 14 09:43:30.102007 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096782    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 14 09:43:30.102007 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096788    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 14 09:43:30.102007 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096792    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 14 09:43:30.102007 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096797    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 14 09:43:30.102007 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096802    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 14 09:43:30.102007 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096806    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 14 09:43:30.102007 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096811    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 14 09:43:30.102007 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096816    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 14 09:43:30.102007 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096821    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 14 09:43:30.102007 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096825    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 14 09:43:30.102007 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096830    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 14 09:43:30.102007 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096835    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 14 09:43:30.102007 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096841    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 14 09:43:30.102007 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096845    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 14 09:43:30.102007 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096849    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 14 09:43:30.102474 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096854    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 14 09:43:30.102474 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096858    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 14 09:43:30.102474 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096862    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 14 09:43:30.102474 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096866    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 14 09:43:30.102474 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096870    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 14 09:43:30.102474 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096875    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 14 09:43:30.102474 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096879    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 14 09:43:30.102474 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096883    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 14 09:43:30.102474 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096887    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 14 09:43:30.102474 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096892    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 14 09:43:30.102474 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096896    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 14 09:43:30.102474 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096900    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 14 09:43:30.102474 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096904    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 14 09:43:30.102474 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096908    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 14 09:43:30.102474 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096912    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 14 09:43:30.102474 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096918    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 14 09:43:30.102474 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096923    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 14 09:43:30.102474 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096928    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 14 09:43:30.102474 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096932    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 14 09:43:30.102474 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096937    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 14 09:43:30.103021 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096942    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 14 09:43:30.103021 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096946    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 14 09:43:30.103021 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096950    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 14 09:43:30.103021 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096955    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 14 09:43:30.103021 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096960    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 14 09:43:30.103021 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096964    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 14 09:43:30.103021 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096969    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 14 09:43:30.103021 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096973    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 14 09:43:30.103021 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096977    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 14 09:43:30.103021 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096981    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 14 09:43:30.103021 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096987    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 14 09:43:30.103021 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096991    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 14 09:43:30.103021 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.096996    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 14 09:43:30.103021 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097000    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 14 09:43:30.103021 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.097009    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 14 09:43:30.103411 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097206    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 14 09:43:30.103411 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097215    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 14 09:43:30.103411 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097221    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 14 09:43:30.103411 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097225    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 14 09:43:30.103411 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097230    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 14 09:43:30.103411 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097234    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 14 09:43:30.103411 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097238    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 14 09:43:30.103411 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097243    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 14 09:43:30.103411 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097247    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 14 09:43:30.103411 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097252    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 14 09:43:30.103411 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097256    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 14 09:43:30.103411 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097260    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 14 09:43:30.103411 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097265    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 14 09:43:30.103411 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097269    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 14 09:43:30.103411 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097273    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 14 09:43:30.103411 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097278    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 14 09:43:30.103411 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097283    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 14 09:43:30.103411 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097287    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 14 09:43:30.103411 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097291    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 14 09:43:30.103411 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097296    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 14 09:43:30.103929 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097300    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 14 09:43:30.103929 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097305    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 14 09:43:30.103929 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097310    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 14 09:43:30.103929 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097314    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 14 09:43:30.103929 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097319    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 14 09:43:30.103929 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097323    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 14 09:43:30.103929 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097327    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 14 09:43:30.103929 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097331    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 14 09:43:30.103929 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097336    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 14 09:43:30.103929 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097341    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 14 09:43:30.103929 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097345    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 14 09:43:30.103929 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097349    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 14 09:43:30.103929 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097354    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 14 09:43:30.103929 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097358    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 14 09:43:30.103929 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097362    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 14 09:43:30.103929 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097367    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 14 09:43:30.103929 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097371    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 14 09:43:30.103929 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097375    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 14 09:43:30.103929 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097380    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 14 09:43:30.103929 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097384    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 14 09:43:30.104425 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097388    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 14 09:43:30.104425 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097392    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 14 09:43:30.104425 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097397    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 14 09:43:30.104425 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097401    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 14 09:43:30.104425 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097405    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 14 09:43:30.104425 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097409    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 14 09:43:30.104425 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097414    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 14 09:43:30.104425 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097418    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 14 09:43:30.104425 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097422    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 14 09:43:30.104425 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097427    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 14 09:43:30.104425 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097432    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 14 09:43:30.104425 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097436    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 14 09:43:30.104425 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097440    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 14 09:43:30.104425 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097445    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 14 09:43:30.104425 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097450    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 14 09:43:30.104425 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097456    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 14 09:43:30.104425 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097462    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 14 09:43:30.104425 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097466    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 14 09:43:30.104425 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097470    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 14 09:43:30.104936 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097476    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 14 09:43:30.104936 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097480    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 14 09:43:30.104936 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097485    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 14 09:43:30.104936 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097490    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 14 09:43:30.104936 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097494    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 14 09:43:30.104936 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097498    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 14 09:43:30.104936 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097504    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 14 09:43:30.104936 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097510    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 14 09:43:30.104936 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097515    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 14 09:43:30.104936 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097520    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 14 09:43:30.104936 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097524    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 14 09:43:30.104936 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097528    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 14 09:43:30.104936 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097533    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 14 09:43:30.104936 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097537    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 14 09:43:30.104936 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097541    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 14 09:43:30.104936 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097545    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 14 09:43:30.104936 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097549    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 14 09:43:30.104936 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097554    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 14 09:43:30.104936 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097558    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 14 09:43:30.105441 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097562    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 14 09:43:30.105441 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097566    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 14 09:43:30.105441 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097571    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 14 09:43:30.105441 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097575    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 14 09:43:30.105441 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097579    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 14 09:43:30.105441 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097583    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 14 09:43:30.105441 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097587    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 14 09:43:30.105441 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097592    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 14 09:43:30.105441 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097596    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 14 09:43:30.105441 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097601    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 14 09:43:30.105441 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097605    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 14 09:43:30.105441 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097609    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 14 09:43:30.105441 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097613    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 14 09:43:30.105441 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097617    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 14 09:43:30.105441 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097622    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 14 09:43:30.105441 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097628    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 14 09:43:30.105441 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097635    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 14 09:43:30.105441 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097640    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 14 09:43:30.105441 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097645    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 14 09:43:30.105441 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097649    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 14 09:43:30.106280 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:30.097653    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 14 09:43:30.106280 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.097662    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 14 09:43:30.106280 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.097914    2563 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 14 09:43:30.106280 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.100768    2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 14 09:43:30.106280 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.101714    2563 server.go:1019] "Starting client certificate rotation"
Sep 14 09:43:30.106280 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.101818    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 14 09:43:30.106280 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.102472    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 14 09:43:30.107811 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.107780    2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 14 09:43:30.112638 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.112612    2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 14 09:43:30.127004 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.126975    2563 log.go:25] "Validated CRI v1 runtime API"
Sep 14 09:43:30.130164 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.130143    2563 log.go:25] "Validated CRI v1 image API"
Sep 14 09:43:30.131257 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.131236    2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 14 09:43:30.135196 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.135163    2563 fs.go:135] Filesystem UUIDs: map[0d1ff808-e5ae-4a5a-946f-8dbdf5ac0998:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 d97328aa-025c-485d-952a-f640dace2e17:/dev/nvme0n1p4]
Sep 14 09:43:30.135196 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.135190    2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 14 09:43:30.138540 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.138511    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 14 09:43:30.142180 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.142025    2563 manager.go:216] Machine: {Timestamp:2026-09-14 09:43:30.140858689 +0000 UTC m=+0.339843775 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3105059 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22c1503fd6f6443ad8c5d3b57bfc93 SystemUUID:ec22c150-3fd6-f644-3ad8-c5d3b57bfc93 BootID:39a9d46a-30ea-4c5a-b246-77d12dbf6d1b Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d3:f8:e8:03 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d3:f8:e8:03 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ae:ff:9f:1b:e0:2d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 14 09:43:30.142180 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.142156    2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 14 09:43:30.142412 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.142357    2563 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 14 09:43:30.143488 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.143456    2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 14 09:43:30.143647 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.143490    2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-4.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 14 09:43:30.144580 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.144566    2563 topology_manager.go:138] "Creating topology manager with none policy"
Sep 14 09:43:30.144619 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.144582    2563 container_manager_linux.go:309] "Creating device plugin manager"
Sep 14 09:43:30.144619 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.144595    2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 14 09:43:30.145145 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.145132    2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 14 09:43:30.145911 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.145899    2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 14 09:43:30.149624 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.149606    2563 state_mem.go:36] "Initialized new in-memory state store"
Sep 14 09:43:30.149785 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.149774    2563 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 14 09:43:30.149834 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.149808    2563 kubelet.go:486] "Attempting to sync node with API server"
Sep 14 09:43:30.149834 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.149820    2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 14 09:43:30.149898 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.149836    2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 14 09:43:30.149898 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.149847    2563 kubelet.go:398] "Adding apiserver pod source"
Sep 14 09:43:30.149898 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.149857    2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 14 09:43:30.150261 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.150247    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 14 09:43:30.150261 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.150267    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 14 09:43:30.154211 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.154180    2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 14 09:43:30.157057 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.157031    2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 14 09:43:30.157192 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.157067    2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 14 09:43:30.158606 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.158585    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 14 09:43:30.158606 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.158605    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 14 09:43:30.158606 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.158612    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 14 09:43:30.158802 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.158617    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 14 09:43:30.158802 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.158626    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 14 09:43:30.158802 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.158633    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 14 09:43:30.158802 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.158652    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 14 09:43:30.158802 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.158658    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 14 09:43:30.158802 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.158665    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 14 09:43:30.158802 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.158671    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 14 09:43:30.158802 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.158680    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 14 09:43:30.158802 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.158685    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 14 09:43:30.158802 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.158721    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 14 09:43:30.158802 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.158726    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 14 09:43:30.162540 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.162521    2563 server.go:1268] "Started kubelet"
Sep 14 09:43:30.162639 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.162607    2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 14 09:43:30.162703 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.162621    2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 14 09:43:30.162877 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.162855    2563 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 14 09:43:30.163460 ip-10-0-132-4 systemd[1]: Started Kubernetes Kubelet.
Sep 14 09:43:30.164247 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.163973    2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 14 09:43:30.164870 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.164843    2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-4.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 14 09:43:30.164954 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:30.164894    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 14 09:43:30.165089 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:30.165058    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-4.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 14 09:43:30.165539 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.165525    2563 server.go:310] "Adding debug handlers to kubelet server"
Sep 14 09:43:30.168229 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:30.168189    2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 14 09:43:30.168406 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.168390    2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 14 09:43:30.168492 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.168404    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 14 09:43:30.168578 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:30.168563    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-4.ec2.internal\" not found"
Sep 14 09:43:30.169003 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.168880    2563 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 14 09:43:30.169003 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.168896    2563 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 14 09:43:30.169003 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.168878    2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 14 09:43:30.169197 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.169022    2563 reconstruct.go:97] "Volume reconstruction finished"
Sep 14 09:43:30.169197 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.169052    2563 reconciler.go:29] "Reconciler: start to sync state"
Sep 14 09:43:30.169314 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.169292    2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 14 09:43:30.169314 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.169313    2563 factory.go:55] Registering systemd factory
Sep 14 09:43:30.169314 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.169321    2563 factory.go:223] Registration of the systemd container factory successfully
Sep 14 09:43:30.169600 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.169581    2563 factory.go:153] Registering CRI-O factory
Sep 14 09:43:30.169600 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.169595    2563 factory.go:223] Registration of the crio container factory successfully
Sep 14 09:43:30.169780 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.169612    2563 factory.go:103] Registering Raw factory
Sep 14 09:43:30.169780 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.169625    2563 manager.go:1196] Started watching for new ooms in manager
Sep 14 09:43:30.170036 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.170021    2563 manager.go:349] Starting recovery of all containers
Sep 14 09:43:30.170957 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:30.170011    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-4.ec2.internal.18d526baad465af9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-4.ec2.internal,UID:ip-10-0-132-4.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-4.ec2.internal,},FirstTimestamp:2026-09-14 09:43:30.162481913 +0000 UTC m=+0.361467000,LastTimestamp:2026-09-14 09:43:30.162481913 +0000 UTC m=+0.361467000,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-4.ec2.internal,}"
Sep 14 09:43:30.177016 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.176958    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 14 09:43:30.179292 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:30.178952    2563 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-4.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 14 09:43:30.179466 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:30.179435    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 14 09:43:30.180260 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.180214    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cxdxz"
Sep 14 09:43:30.180826 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.180807    2563 manager.go:354] Recovery completed
Sep 14 09:43:30.187379 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.187354    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cxdxz"
Sep 14 09:43:30.188152 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.188133    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 14 09:43:30.190054 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.190035    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeHasSufficientMemory"
Sep 14 09:43:30.190139 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.190069    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeHasNoDiskPressure"
Sep 14 09:43:30.190139 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.190085    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeHasSufficientPID"
Sep 14 09:43:30.190629 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.190614    2563 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 14 09:43:30.190629 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.190628    2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 14 09:43:30.190719 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.190644    2563 state_mem.go:36] "Initialized new in-memory state store"
Sep 14 09:43:30.190719 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.190700    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 14 09:43:30.191489 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.191475    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeHasSufficientMemory"
Sep 14 09:43:30.191590 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.191500    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeHasNoDiskPressure"
Sep 14 09:43:30.191590 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.191511    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeHasSufficientPID"
Sep 14 09:43:30.192502 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:30.192427    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-4.ec2.internal.18d526baaeeb1096  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-4.ec2.internal,UID:ip-10-0-132-4.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-4.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-4.ec2.internal,},FirstTimestamp:2026-09-14 09:43:30.190053526 +0000 UTC m=+0.389038612,LastTimestamp:2026-09-14 09:43:30.190053526 +0000 UTC m=+0.389038612,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-4.ec2.internal,}"
Sep 14 09:43:30.192614 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.192602    2563 policy_none.go:49] "None policy: Start"
Sep 14 09:43:30.192648 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.192621    2563 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 14 09:43:30.192648 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.192632    2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 14 09:43:30.194730 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.194714    2563 policy_none.go:47] "Start"
Sep 14 09:43:30.244520 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.244493    2563 manager.go:337] "Starting Device Plugin manager"
Sep 14 09:43:30.255972 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:30.244573    2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 14 09:43:30.255972 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.244589    2563 server.go:87] "Starting device plugin registration server"
Sep 14 09:43:30.255972 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.244934    2563 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 14 09:43:30.255972 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.244946    2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 14 09:43:30.255972 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.245039    2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 14 09:43:30.255972 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.245135    2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 14 09:43:30.255972 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.245147    2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 14 09:43:30.255972 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:30.245666    2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 14 09:43:30.255972 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:30.245725    2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-4.ec2.internal\" not found"
Sep 14 09:43:30.315942 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.315905    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 14 09:43:30.316098 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.315974    2563 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 14 09:43:30.316098 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.315998    2563 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 14 09:43:30.316098 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:30.316038    2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 14 09:43:30.319060 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.319023    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 14 09:43:30.345718 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.345602    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 14 09:43:30.346620 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.346603    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeHasSufficientMemory"
Sep 14 09:43:30.346737 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.346633    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeHasNoDiskPressure"
Sep 14 09:43:30.346737 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.346646    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeHasSufficientPID"
Sep 14 09:43:30.346737 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.346671    2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-4.ec2.internal"
Sep 14 09:43:30.355226 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.355201    2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-4.ec2.internal"
Sep 14 09:43:30.355285 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:30.355236    2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-4.ec2.internal\": node \"ip-10-0-132-4.ec2.internal\" not found"
Sep 14 09:43:30.386246 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:30.386215    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-4.ec2.internal\" not found"
Sep 14 09:43:30.416856 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.416822    2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-4.ec2.internal"]
Sep 14 09:43:30.417021 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.416914    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 14 09:43:30.418441 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.418420    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeHasSufficientMemory"
Sep 14 09:43:30.418519 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.418460    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeHasNoDiskPressure"
Sep 14 09:43:30.418519 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.418472    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeHasSufficientPID"
Sep 14 09:43:30.420808 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.420793    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 14 09:43:30.420931 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.420912    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal"
Sep 14 09:43:30.420996 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.420950    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 14 09:43:30.422280 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.422262    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeHasSufficientMemory"
Sep 14 09:43:30.422280 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.422271    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeHasSufficientMemory"
Sep 14 09:43:30.422438 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.422292    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeHasNoDiskPressure"
Sep 14 09:43:30.422438 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.422307    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeHasSufficientPID"
Sep 14 09:43:30.422438 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.422293    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeHasNoDiskPressure"
Sep 14 09:43:30.422438 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.422376    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeHasSufficientPID"
Sep 14 09:43:30.424679 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.424663    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-4.ec2.internal"
Sep 14 09:43:30.424804 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.424702    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 14 09:43:30.425446 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.425430    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeHasSufficientMemory"
Sep 14 09:43:30.425538 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.425458    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeHasNoDiskPressure"
Sep 14 09:43:30.425538 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.425467    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeHasSufficientPID"
Sep 14 09:43:30.448592 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:30.448561    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-4.ec2.internal\" not found" node="ip-10-0-132-4.ec2.internal"
Sep 14 09:43:30.453155 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:30.453132    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-4.ec2.internal\" not found" node="ip-10-0-132-4.ec2.internal"
Sep 14 09:43:30.470209 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.470170    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/53b749d177a0ad18e520172c7d6ddac0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal\" (UID: \"53b749d177a0ad18e520172c7d6ddac0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal"
Sep 14 09:43:30.470209 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.470212    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/53b749d177a0ad18e520172c7d6ddac0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal\" (UID: \"53b749d177a0ad18e520172c7d6ddac0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal"
Sep 14 09:43:30.470408 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.470239    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/812604c7944c05f98e0873536f91322d-config\") pod \"kube-apiserver-proxy-ip-10-0-132-4.ec2.internal\" (UID: \"812604c7944c05f98e0873536f91322d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-4.ec2.internal"
Sep 14 09:43:30.486998 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:30.486961    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-4.ec2.internal\" not found"
Sep 14 09:43:30.570876 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.570835    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/812604c7944c05f98e0873536f91322d-config\") pod \"kube-apiserver-proxy-ip-10-0-132-4.ec2.internal\" (UID: \"812604c7944c05f98e0873536f91322d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-4.ec2.internal"
Sep 14 09:43:30.570876 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.570883    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/53b749d177a0ad18e520172c7d6ddac0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal\" (UID: \"53b749d177a0ad18e520172c7d6ddac0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal"
Sep 14 09:43:30.571087 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.570906    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/53b749d177a0ad18e520172c7d6ddac0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal\" (UID: \"53b749d177a0ad18e520172c7d6ddac0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal"
Sep 14 09:43:30.571087 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.570908    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/812604c7944c05f98e0873536f91322d-config\") pod \"kube-apiserver-proxy-ip-10-0-132-4.ec2.internal\" (UID: \"812604c7944c05f98e0873536f91322d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-4.ec2.internal"
Sep 14 09:43:30.571087 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.570945    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/53b749d177a0ad18e520172c7d6ddac0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal\" (UID: \"53b749d177a0ad18e520172c7d6ddac0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal"
Sep 14 09:43:30.571087 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.570970    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/53b749d177a0ad18e520172c7d6ddac0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal\" (UID: \"53b749d177a0ad18e520172c7d6ddac0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal"
Sep 14 09:43:30.588030 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:30.587998    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-4.ec2.internal\" not found"
Sep 14 09:43:30.688380 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:30.688313    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-4.ec2.internal\" not found"
Sep 14 09:43:30.751526 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.751488    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal"
Sep 14 09:43:30.756441 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:30.756416    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-4.ec2.internal"
Sep 14 09:43:30.788830 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:30.788783    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-4.ec2.internal\" not found"
Sep 14 09:43:30.889334 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:30.889294    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-4.ec2.internal\" not found"
Sep 14 09:43:30.989897 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:30.989821    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-4.ec2.internal\" not found"
Sep 14 09:43:31.090506 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:31.090466    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-4.ec2.internal\" not found"
Sep 14 09:43:31.101655 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:31.101626    2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 14 09:43:31.101823 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:31.101805    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 14 09:43:31.161762 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:31.161563    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 14 09:43:31.191361 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:31.191321    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-4.ec2.internal\" not found"
Sep 14 09:43:31.192451 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:31.192414    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-13 09:38:31 +0000 UTC" deadline="2028-05-02 04:52:17.014584606 +0000 UTC"
Sep 14 09:43:31.192451 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:31.192445    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14299h8m45.82214278s"
Sep 14 09:43:31.292217 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:31.292135    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-4.ec2.internal\" not found"
Sep 14 09:43:31.331865 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:31.331825    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53b749d177a0ad18e520172c7d6ddac0.slice/crio-cc7858d9f12b827f6ae407d2b3995e37d9fdcbf7508aa91878e50c6acc9f43e4 WatchSource:0}: Error finding container cc7858d9f12b827f6ae407d2b3995e37d9fdcbf7508aa91878e50c6acc9f43e4: Status 404 returned error can't find the container with id cc7858d9f12b827f6ae407d2b3995e37d9fdcbf7508aa91878e50c6acc9f43e4
Sep 14 09:43:31.332146 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:31.332126    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod812604c7944c05f98e0873536f91322d.slice/crio-5ab647349c219fe21c2cfa151e21dd50928af9db88c3e1dd33136b20b7e3b854 WatchSource:0}: Error finding container 5ab647349c219fe21c2cfa151e21dd50928af9db88c3e1dd33136b20b7e3b854: Status 404 returned error can't find the container with id 5ab647349c219fe21c2cfa151e21dd50928af9db88c3e1dd33136b20b7e3b854
Sep 14 09:43:31.337939 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:31.337919    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 14 09:43:31.393117 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:31.393077    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-4.ec2.internal\" not found"
Sep 14 09:43:31.493617 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:31.493581    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-4.ec2.internal\" not found"
Sep 14 09:43:31.594177 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:31.594137    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-4.ec2.internal\" not found"
Sep 14 09:43:31.641351 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:31.641311    2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 14 09:43:31.694871 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:31.694826    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-4.ec2.internal\" not found"
Sep 14 09:43:31.713665 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:31.713622    2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 14 09:43:31.769494 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:31.769451    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal"
Sep 14 09:43:31.783448 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:31.783409    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 14 09:43:31.784425 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:31.784394    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-4.ec2.internal"
Sep 14 09:43:31.793370 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:31.793324    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 14 09:43:32.151193 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.151156    2563 apiserver.go:52] "Watching apiserver"
Sep 14 09:43:32.161315 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.161279    2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 14 09:43:32.162518 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.162446    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-7tm8l","openshift-multus/network-metrics-daemon-6l89k","openshift-ovn-kubernetes/ovnkube-node-hdclt","kube-system/global-pull-secret-syncer-krg4v","kube-system/konnectivity-agent-cskmj","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjwpq","openshift-image-registry/node-ca-mq9n4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal","openshift-multus/multus-tcvsm","openshift-network-diagnostics/network-check-target-8zcts","openshift-network-operator/iptables-alerter-cgxf2","kube-system/kube-apiserver-proxy-ip-10-0-132-4.ec2.internal","openshift-cluster-node-tuning-operator/tuned-kb4pz","openshift-dns/node-resolver-f5jts"]
Sep 14 09:43:32.165129 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.165091    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mq9n4"
Sep 14 09:43:32.167438 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.167406    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kb4pz"
Sep 14 09:43:32.169183 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.169159    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 14 09:43:32.170213 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.170176    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hdclt"
Sep 14 09:43:32.172009 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.171818    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wh4fc\""
Sep 14 09:43:32.172155 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.172116    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 14 09:43:32.172350 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.172330    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 14 09:43:32.172558 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.172543    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 14 09:43:32.172847 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.172827    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krg4v"
Sep 14 09:43:32.172937 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:32.172909    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-krg4v" podUID="97284aa2-9a80-495f-8b99-2ff7cec913da"
Sep 14 09:43:32.174525 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.174489    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 14 09:43:32.175408 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.174817    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-585bf\""
Sep 14 09:43:32.175408 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.175122    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 14 09:43:32.175408 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.175151    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 14 09:43:32.175408 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.175294    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 14 09:43:32.175408 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.175306    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 14 09:43:32.175637 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.175543    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 14 09:43:32.175637 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.175599    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 14 09:43:32.175637 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.175621    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cskmj"
Sep 14 09:43:32.175990 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.175848    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 14 09:43:32.175990 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.175944    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mgvd9\""
Sep 14 09:43:32.177581 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.177558    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjwpq"
Sep 14 09:43:32.178201 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.177993    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 14 09:43:32.179013 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.178983    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 14 09:43:32.179510 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.179492    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-lrgk4\""
Sep 14 09:43:32.179674 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.179647    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/07d5e1df-7d96-401d-b606-245e4b1f3ae5-serviceca\") pod \"node-ca-mq9n4\" (UID: \"07d5e1df-7d96-401d-b606-245e4b1f3ae5\") " pod="openshift-image-registry/node-ca-mq9n4"
Sep 14 09:43:32.179787 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.179676    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/da9fcef7-654c-4140-bb22-abd2ece6e0c1-etc-tuned\") pod \"tuned-kb4pz\" (UID: \"da9fcef7-654c-4140-bb22-abd2ece6e0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-kb4pz"
Sep 14 09:43:32.179787 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.179721    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dba3fdc7-1cc0-451e-9359-f597a2fb65a3-ovn-node-metrics-cert\") pod \"ovnkube-node-hdclt\" (UID: \"dba3fdc7-1cc0-451e-9359-f597a2fb65a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdclt"
Sep 14 09:43:32.179787 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.179745    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/da9fcef7-654c-4140-bb22-abd2ece6e0c1-etc-systemd\") pod \"tuned-kb4pz\" (UID: \"da9fcef7-654c-4140-bb22-abd2ece6e0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-kb4pz"
Sep 14 09:43:32.179787 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.179770    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqczc\" (UniqueName: \"kubernetes.io/projected/da9fcef7-654c-4140-bb22-abd2ece6e0c1-kube-api-access-tqczc\") pod \"tuned-kb4pz\" (UID: \"da9fcef7-654c-4140-bb22-abd2ece6e0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-kb4pz"
Sep 14 09:43:32.179990 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.179814    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dba3fdc7-1cc0-451e-9359-f597a2fb65a3-run-systemd\") pod \"ovnkube-node-hdclt\" (UID: \"dba3fdc7-1cc0-451e-9359-f597a2fb65a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdclt"
Sep 14 09:43:32.179990 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.179847    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dba3fdc7-1cc0-451e-9359-f597a2fb65a3-host-run-ovn-kubernetes\") pod \"ovnkube-node-hdclt\" (UID: \"dba3fdc7-1cc0-451e-9359-f597a2fb65a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdclt"
Sep 14 09:43:32.179990 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.179880    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/da9fcef7-654c-4140-bb22-abd2ece6e0c1-tmp\") pod \"tuned-kb4pz\" (UID: \"da9fcef7-654c-4140-bb22-abd2ece6e0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-kb4pz"
Sep 14 09:43:32.179990 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.179904    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dba3fdc7-1cc0-451e-9359-f597a2fb65a3-host-cni-netd\") pod \"ovnkube-node-hdclt\" (UID: \"dba3fdc7-1cc0-451e-9359-f597a2fb65a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdclt"
Sep 14 09:43:32.179990 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.179929    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dba3fdc7-1cc0-451e-9359-f597a2fb65a3-env-overrides\") pod \"ovnkube-node-hdclt\" (UID: \"dba3fdc7-1cc0-451e-9359-f597a2fb65a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdclt"
Sep 14 09:43:32.179990 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.179952    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/da9fcef7-654c-4140-bb22-abd2ece6e0c1-etc-sysconfig\") pod \"tuned-kb4pz\" (UID: \"da9fcef7-654c-4140-bb22-abd2ece6e0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-kb4pz"
Sep 14 09:43:32.179990 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.179974    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/da9fcef7-654c-4140-bb22-abd2ece6e0c1-host\") pod \"tuned-kb4pz\" (UID: \"da9fcef7-654c-4140-bb22-abd2ece6e0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-kb4pz"
Sep 14 09:43:32.180313 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.179996    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dba3fdc7-1cc0-451e-9359-f597a2fb65a3-host-run-netns\") pod \"ovnkube-node-hdclt\" (UID: \"dba3fdc7-1cc0-451e-9359-f597a2fb65a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdclt"
Sep 14 09:43:32.180313 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.180027    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dba3fdc7-1cc0-451e-9359-f597a2fb65a3-var-lib-openvswitch\") pod \"ovnkube-node-hdclt\" (UID: \"dba3fdc7-1cc0-451e-9359-f597a2fb65a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdclt"
Sep 14 09:43:32.180313 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.180052    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dba3fdc7-1cc0-451e-9359-f597a2fb65a3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hdclt\" (UID: \"dba3fdc7-1cc0-451e-9359-f597a2fb65a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdclt"
Sep 14 09:43:32.180313 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.180077    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dba3fdc7-1cc0-451e-9359-f597a2fb65a3-ovnkube-config\") pod \"ovnkube-node-hdclt\" (UID: \"dba3fdc7-1cc0-451e-9359-f597a2fb65a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdclt"
Sep 14 09:43:32.180313 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.180116    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5s6m\" (UniqueName: \"kubernetes.io/projected/07d5e1df-7d96-401d-b606-245e4b1f3ae5-kube-api-access-x5s6m\") pod \"node-ca-mq9n4\" (UID: \"07d5e1df-7d96-401d-b606-245e4b1f3ae5\") " pod="openshift-image-registry/node-ca-mq9n4"
Sep 14 09:43:32.180313 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.180151    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/da9fcef7-654c-4140-bb22-abd2ece6e0c1-etc-sysctl-d\") pod \"tuned-kb4pz\" (UID: \"da9fcef7-654c-4140-bb22-abd2ece6e0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-kb4pz"
Sep 14 09:43:32.180313 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.180168    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/da9fcef7-654c-4140-bb22-abd2ece6e0c1-etc-sysctl-conf\") pod \"tuned-kb4pz\" (UID: \"da9fcef7-654c-4140-bb22-abd2ece6e0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-kb4pz"
Sep 14 09:43:32.180313 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.180198    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dba3fdc7-1cc0-451e-9359-f597a2fb65a3-etc-openvswitch\") pod \"ovnkube-node-hdclt\" (UID: \"dba3fdc7-1cc0-451e-9359-f597a2fb65a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdclt"
Sep 14 09:43:32.180313 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.180227    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/da9fcef7-654c-4140-bb22-abd2ece6e0c1-etc-kubernetes\") pod \"tuned-kb4pz\" (UID: \"da9fcef7-654c-4140-bb22-abd2ece6e0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-kb4pz"
Sep 14 09:43:32.180313 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.180243    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/da9fcef7-654c-4140-bb22-abd2ece6e0c1-lib-modules\") pod \"tuned-kb4pz\" (UID: \"da9fcef7-654c-4140-bb22-abd2ece6e0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-kb4pz"
Sep 14 09:43:32.180313 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.180256    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dba3fdc7-1cc0-451e-9359-f597a2fb65a3-run-openvswitch\") pod \"ovnkube-node-hdclt\" (UID: \"dba3fdc7-1cc0-451e-9359-f597a2fb65a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdclt"
Sep 14 09:43:32.180313 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.180289    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dba3fdc7-1cc0-451e-9359-f597a2fb65a3-run-ovn\") pod \"ovnkube-node-hdclt\" (UID: \"dba3fdc7-1cc0-451e-9359-f597a2fb65a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdclt"
Sep 14 09:43:32.180892 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.180332    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dba3fdc7-1cc0-451e-9359-f597a2fb65a3-node-log\") pod \"ovnkube-node-hdclt\" (UID: \"dba3fdc7-1cc0-451e-9359-f597a2fb65a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdclt"
Sep 14 09:43:32.180892 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.180364    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn7bs\" (UniqueName: \"kubernetes.io/projected/dba3fdc7-1cc0-451e-9359-f597a2fb65a3-kube-api-access-vn7bs\") pod \"ovnkube-node-hdclt\" (UID: \"dba3fdc7-1cc0-451e-9359-f597a2fb65a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdclt"
Sep 14 09:43:32.180892 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.180404    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/da9fcef7-654c-4140-bb22-abd2ece6e0c1-etc-modprobe-d\") pod \"tuned-kb4pz\" (UID: \"da9fcef7-654c-4140-bb22-abd2ece6e0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-kb4pz"
Sep 14 09:43:32.180892 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.180428    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/da9fcef7-654c-4140-bb22-abd2ece6e0c1-sys\") pod \"tuned-kb4pz\" (UID: \"da9fcef7-654c-4140-bb22-abd2ece6e0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-kb4pz"
Sep 14 09:43:32.180892 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.180451    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dba3fdc7-1cc0-451e-9359-f597a2fb65a3-host-kubelet\") pod \"ovnkube-node-hdclt\" (UID: \"dba3fdc7-1cc0-451e-9359-f597a2fb65a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdclt"
Sep 14 09:43:32.180892 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.180490    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dba3fdc7-1cc0-451e-9359-f597a2fb65a3-systemd-units\") pod \"ovnkube-node-hdclt\" (UID: \"dba3fdc7-1cc0-451e-9359-f597a2fb65a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdclt"
Sep 14 09:43:32.180892 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.180534    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/da9fcef7-654c-4140-bb22-abd2ece6e0c1-run\") pod \"tuned-kb4pz\" (UID: \"da9fcef7-654c-4140-bb22-abd2ece6e0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-kb4pz"
Sep 14 09:43:32.180892 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.180567    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/da9fcef7-654c-4140-bb22-abd2ece6e0c1-var-lib-kubelet\") pod \"tuned-kb4pz\" (UID: \"da9fcef7-654c-4140-bb22-abd2ece6e0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-kb4pz"
Sep 14 09:43:32.180892 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.180594    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dba3fdc7-1cc0-451e-9359-f597a2fb65a3-host-slash\") pod \"ovnkube-node-hdclt\" (UID: \"dba3fdc7-1cc0-451e-9359-f597a2fb65a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdclt"
Sep 14 09:43:32.180892 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.180619    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dba3fdc7-1cc0-451e-9359-f597a2fb65a3-log-socket\") pod \"ovnkube-node-hdclt\" (UID: \"dba3fdc7-1cc0-451e-9359-f597a2fb65a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdclt"
Sep 14 09:43:32.180892 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.180648    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dba3fdc7-1cc0-451e-9359-f597a2fb65a3-host-cni-bin\") pod \"ovnkube-node-hdclt\" (UID: \"dba3fdc7-1cc0-451e-9359-f597a2fb65a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdclt"
Sep 14 09:43:32.180892 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.180676    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dba3fdc7-1cc0-451e-9359-f597a2fb65a3-ovnkube-script-lib\") pod \"ovnkube-node-hdclt\" (UID: \"dba3fdc7-1cc0-451e-9359-f597a2fb65a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdclt"
Sep 14 09:43:32.180892 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.180724    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/07d5e1df-7d96-401d-b606-245e4b1f3ae5-host\") pod \"node-ca-mq9n4\" (UID: \"07d5e1df-7d96-401d-b606-245e4b1f3ae5\") " pod="openshift-image-registry/node-ca-mq9n4"
Sep 14 09:43:32.180892 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.180806    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tcvsm"
Sep 14 09:43:32.181573 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.181507    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 14 09:43:32.181673 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.181653    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 14 09:43:32.181759 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.181680    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 14 09:43:32.181759 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.181708    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9jhfc\""
Sep 14 09:43:32.183857 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.183837    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 14 09:43:32.184514 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.184152    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 14 09:43:32.184514 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.184207    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 14 09:43:32.184514 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.184240    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 14 09:43:32.184514 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.184279    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7tm8l"
Sep 14 09:43:32.184514 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.184478    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-8zf54\""
Sep 14 09:43:32.187233 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.186977    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4whkh\""
Sep 14 09:43:32.187233 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.186990    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6l89k"
Sep 14 09:43:32.187233 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:32.187067    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6l89k" podUID="d956c018-b9cb-4c19-8f08-16652611d8c3"
Sep 14 09:43:32.187660 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.187640    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 14 09:43:32.187916 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.187899    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 14 09:43:32.188500 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.188477    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 14 09:43:32.189563 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.189543    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8zcts"
Sep 14 09:43:32.189670 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:32.189629    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8zcts" podUID="c41422ae-06c2-492c-9dbf-82e10222589a"
Sep 14 09:43:32.192989 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.192523    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cgxf2"
Sep 14 09:43:32.195080 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.195054    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-f5jts"
Sep 14 09:43:32.197578 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.197552    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 14 09:43:32.197909 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.197885    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nfpxs\""
Sep 14 09:43:32.198012 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.197940    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 14 09:43:32.198131 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.198112    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 14 09:43:32.199631 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.199606    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 14 09:43:32.199749 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.199717    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 14 09:43:32.200213 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.200191    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bndtx\""
Sep 14 09:43:32.228290 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.228259    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-fs7qv"
Sep 14 09:43:32.236957 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.236925    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-fs7qv"
Sep 14 09:43:32.270453 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.270423    2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 14 09:43:32.280985 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.280950    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dba3fdc7-1cc0-451e-9359-f597a2fb65a3-host-run-netns\") pod \"ovnkube-node-hdclt\" (UID: \"dba3fdc7-1cc0-451e-9359-f597a2fb65a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdclt"
Sep 14 09:43:32.280985 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.280986    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dba3fdc7-1cc0-451e-9359-f597a2fb65a3-var-lib-openvswitch\") pod \"ovnkube-node-hdclt\" (UID: \"dba3fdc7-1cc0-451e-9359-f597a2fb65a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdclt"
Sep 14 09:43:32.281197 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.281012    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dba3fdc7-1cc0-451e-9359-f597a2fb65a3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hdclt\" (UID: \"dba3fdc7-1cc0-451e-9359-f597a2fb65a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdclt"
Sep 14 09:43:32.281197 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.281064    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dba3fdc7-1cc0-451e-9359-f597a2fb65a3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hdclt\" (UID: \"dba3fdc7-1cc0-451e-9359-f597a2fb65a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdclt"
Sep 14 09:43:32.281197 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.281079    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dba3fdc7-1cc0-451e-9359-f597a2fb65a3-var-lib-openvswitch\") pod \"ovnkube-node-hdclt\" (UID: \"dba3fdc7-1cc0-451e-9359-f597a2fb65a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdclt"
Sep 14 09:43:32.281197 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.281083    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dba3fdc7-1cc0-451e-9359-f597a2fb65a3-ovnkube-config\") pod \"ovnkube-node-hdclt\" (UID: \"dba3fdc7-1cc0-451e-9359-f597a2fb65a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdclt"
Sep 14 09:43:32.281197 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.281144    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/66c47172-17e5-4985-84a8-987c216dd4ec-konnectivity-ca\") pod \"konnectivity-agent-cskmj\" (UID: \"66c47172-17e5-4985-84a8-987c216dd4ec\") " pod="kube-system/konnectivity-agent-cskmj"
Sep 14 09:43:32.281197 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.281163    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1c80666a-ce28-47ef-b790-be0563101bb4-cnibin\") pod \"multus-tcvsm\" (UID: \"1c80666a-ce28-47ef-b790-be0563101bb4\") " pod="openshift-multus/multus-tcvsm"
Sep 14 09:43:32.281197 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.281064    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dba3fdc7-1cc0-451e-9359-f597a2fb65a3-host-run-netns\") pod \"ovnkube-node-hdclt\" (UID: \"dba3fdc7-1cc0-451e-9359-f597a2fb65a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdclt"
Sep 14 09:43:32.281197 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.281183    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1c80666a-ce28-47ef-b790-be0563101bb4-cni-binary-copy\") pod \"multus-tcvsm\" (UID: \"1c80666a-ce28-47ef-b790-be0563101bb4\") " pod="openshift-multus/multus-tcvsm"
Sep 14 09:43:32.281197 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.281203    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/da9fcef7-654c-4140-bb22-abd2ece6e0c1-etc-sysctl-conf\") pod \"tuned-kb4pz\" (UID: \"da9fcef7-654c-4140-bb22-abd2ece6e0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-kb4pz"
Sep 14 09:43:32.281552 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.281224    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0a7d1077-b7fb-479d-990d-e35e6e7611bd-registration-dir\") pod \"aws-ebs-csi-driver-node-fjwpq\" (UID: \"0a7d1077-b7fb-479d-990d-e35e6e7611bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjwpq"
Sep 14 09:43:32.281552 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.281241    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1c80666a-ce28-47ef-b790-be0563101bb4-system-cni-dir\") pod \"multus-tcvsm\" (UID: \"1c80666a-ce28-47ef-b790-be0563101bb4\") " pod="openshift-multus/multus-tcvsm"
Sep 14 09:43:32.281552 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.281259    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1c80666a-ce28-47ef-b790-be0563101bb4-host-run-k8s-cni-cncf-io\") pod \"multus-tcvsm\" (UID: \"1c80666a-ce28-47ef-b790-be0563101bb4\") " pod="openshift-multus/multus-tcvsm"
Sep 14 09:43:32.281552 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.281281    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c80666a-ce28-47ef-b790-be0563101bb4-host-var-lib-kubelet\") pod \"multus-tcvsm\" (UID: \"1c80666a-ce28-47ef-b790-be0563101bb4\") " pod="openshift-multus/multus-tcvsm"
Sep 14 09:43:32.281552 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.281301    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1c80666a-ce28-47ef-b790-be0563101bb4-multus-conf-dir\") pod \"multus-tcvsm\" (UID: \"1c80666a-ce28-47ef-b790-be0563101bb4\") " pod="openshift-multus/multus-tcvsm"
Sep 14 09:43:32.281552 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.281316    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/20d9618f-664c-4cae-bcc2-4b00ba45acc6-cni-binary-copy\") pod \"multus-additional-cni-plugins-7tm8l\" (UID: \"20d9618f-664c-4cae-bcc2-4b00ba45acc6\") " pod="openshift-multus/multus-additional-cni-plugins-7tm8l"
Sep 14 09:43:32.281552 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.281344    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/97284aa2-9a80-495f-8b99-2ff7cec913da-kubelet-config\") pod \"global-pull-secret-syncer-krg4v\" (UID: \"97284aa2-9a80-495f-8b99-2ff7cec913da\") " pod="kube-system/global-pull-secret-syncer-krg4v"
Sep 14 09:43:32.281552 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.281375    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/da9fcef7-654c-4140-bb22-abd2ece6e0c1-lib-modules\") pod \"tuned-kb4pz\" (UID: \"da9fcef7-654c-4140-bb22-abd2ece6e0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-kb4pz"
Sep 14 09:43:32.281552 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.281394    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dba3fdc7-1cc0-451e-9359-f597a2fb65a3-run-openvswitch\") pod \"ovnkube-node-hdclt\" (UID: \"dba3fdc7-1cc0-451e-9359-f597a2fb65a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdclt"
Sep 14 09:43:32.281552 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.281436    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/da9fcef7-654c-4140-bb22-abd2ece6e0c1-etc-sysctl-conf\") pod \"tuned-kb4pz\" (UID: \"da9fcef7-654c-4140-bb22-abd2ece6e0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-kb4pz"
Sep 14 09:43:32.281552 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.281443    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dba3fdc7-1cc0-451e-9359-f597a2fb65a3-run-ovn\") pod \"ovnkube-node-hdclt\" (UID: \"dba3fdc7-1cc0-451e-9359-f597a2fb65a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdclt"
Sep 14 09:43:32.281552 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.281464    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dba3fdc7-1cc0-451e-9359-f597a2fb65a3-run-openvswitch\") pod \"ovnkube-node-hdclt\" (UID: \"dba3fdc7-1cc0-451e-9359-f597a2fb65a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdclt"
Sep 14 09:43:32.281552 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.281512    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dba3fdc7-1cc0-451e-9359-f597a2fb65a3-run-ovn\") pod \"ovnkube-node-hdclt\" (UID: \"dba3fdc7-1cc0-451e-9359-f597a2fb65a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdclt"
Sep 14 09:43:32.281552 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.281535    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dba3fdc7-1cc0-451e-9359-f597a2fb65a3-node-log\") pod \"ovnkube-node-hdclt\" (UID: \"dba3fdc7-1cc0-451e-9359-f597a2fb65a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdclt"
Sep 14 09:43:32.281552 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.281530    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/da9fcef7-654c-4140-bb22-abd2ece6e0c1-lib-modules\") pod \"tuned-kb4pz\" (UID: \"da9fcef7-654c-4140-bb22-abd2ece6e0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-kb4pz"
Sep 14 09:43:32.281552 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.281553    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1c80666a-ce28-47ef-b790-be0563101bb4-host-var-lib-cni-multus\") pod \"multus-tcvsm\" (UID: \"1c80666a-ce28-47ef-b790-be0563101bb4\") " pod="openshift-multus/multus-tcvsm"
Sep 14 09:43:32.282123 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.281581    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dba3fdc7-1cc0-451e-9359-f597a2fb65a3-node-log\") pod \"ovnkube-node-hdclt\" (UID: \"dba3fdc7-1cc0-451e-9359-f597a2fb65a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdclt"
Sep 14 09:43:32.282123 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.281571    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/da9fcef7-654c-4140-bb22-abd2ece6e0c1-etc-modprobe-d\") pod \"tuned-kb4pz\" (UID: \"da9fcef7-654c-4140-bb22-abd2ece6e0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-kb4pz"
Sep 14 09:43:32.282123 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.281625    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dba3fdc7-1cc0-451e-9359-f597a2fb65a3-systemd-units\") pod \"ovnkube-node-hdclt\" (UID: \"dba3fdc7-1cc0-451e-9359-f597a2fb65a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdclt"
Sep 14 09:43:32.282123 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.281652    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1c80666a-ce28-47ef-b790-be0563101bb4-multus-socket-dir-parent\") pod \"multus-tcvsm\" (UID: \"1c80666a-ce28-47ef-b790-be0563101bb4\") " pod="openshift-multus/multus-tcvsm"
Sep 14 09:43:32.282123 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.281659    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dba3fdc7-1cc0-451e-9359-f597a2fb65a3-ovnkube-config\") pod \"ovnkube-node-hdclt\" (UID: \"dba3fdc7-1cc0-451e-9359-f597a2fb65a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdclt"
Sep 14 09:43:32.282123 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.281673    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/da9fcef7-654c-4140-bb22-abd2ece6e0c1-etc-modprobe-d\") pod \"tuned-kb4pz\" (UID: \"da9fcef7-654c-4140-bb22-abd2ece6e0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-kb4pz"
Sep 14 09:43:32.282123 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.281677    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1c80666a-ce28-47ef-b790-be0563101bb4-host-run-netns\") pod \"multus-tcvsm\" (UID: \"1c80666a-ce28-47ef-b790-be0563101bb4\") " pod="openshift-multus/multus-tcvsm"
Sep 14 09:43:32.282123 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.281684    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dba3fdc7-1cc0-451e-9359-f597a2fb65a3-systemd-units\") pod \"ovnkube-node-hdclt\" (UID: \"dba3fdc7-1cc0-451e-9359-f597a2fb65a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdclt"
Sep 14 09:43:32.282123 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.281739    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/da9fcef7-654c-4140-bb22-abd2ece6e0c1-var-lib-kubelet\") pod \"tuned-kb4pz\" (UID: \"da9fcef7-654c-4140-bb22-abd2ece6e0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-kb4pz"
Sep 14 09:43:32.282123 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.281756    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dba3fdc7-1cc0-451e-9359-f597a2fb65a3-log-socket\") pod \"ovnkube-node-hdclt\" (UID: \"dba3fdc7-1cc0-451e-9359-f597a2fb65a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdclt"
Sep 14 09:43:32.282123 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.281781    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dba3fdc7-1cc0-451e-9359-f597a2fb65a3-host-cni-bin\") pod \"ovnkube-node-hdclt\" (UID: \"dba3fdc7-1cc0-451e-9359-f597a2fb65a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdclt"
Sep 14 09:43:32.282123 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.281801    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dba3fdc7-1cc0-451e-9359-f597a2fb65a3-ovnkube-script-lib\") pod \"ovnkube-node-hdclt\" (UID: \"dba3fdc7-1cc0-451e-9359-f597a2fb65a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdclt"
Sep 14 09:43:32.282123 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.281808    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dba3fdc7-1cc0-451e-9359-f597a2fb65a3-log-socket\") pod \"ovnkube-node-hdclt\" (UID: \"dba3fdc7-1cc0-451e-9359-f597a2fb65a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdclt"
Sep 14 09:43:32.282123 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.281818    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xtjs\" (UniqueName: \"kubernetes.io/projected/da4b31da-fcbb-4132-bc49-1ecd91bb65bb-kube-api-access-9xtjs\") pod \"node-resolver-f5jts\" (UID: \"da4b31da-fcbb-4132-bc49-1ecd91bb65bb\") " pod="openshift-dns/node-resolver-f5jts"
Sep 14 09:43:32.282123 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.281819    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/da9fcef7-654c-4140-bb22-abd2ece6e0c1-var-lib-kubelet\") pod \"tuned-kb4pz\" (UID: \"da9fcef7-654c-4140-bb22-abd2ece6e0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-kb4pz"
Sep 14 09:43:32.282123 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.281853    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/07d5e1df-7d96-401d-b606-245e4b1f3ae5-host\") pod \"node-ca-mq9n4\" (UID: \"07d5e1df-7d96-401d-b606-245e4b1f3ae5\") " pod="openshift-image-registry/node-ca-mq9n4"
Sep 14 09:43:32.282123 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.281950    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0a7d1077-b7fb-479d-990d-e35e6e7611bd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fjwpq\" (UID: \"0a7d1077-b7fb-479d-990d-e35e6e7611bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjwpq"
Sep 14 09:43:32.282856 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.281953    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/07d5e1df-7d96-401d-b606-245e4b1f3ae5-host\") pod \"node-ca-mq9n4\" (UID: \"07d5e1df-7d96-401d-b606-245e4b1f3ae5\") " pod="openshift-image-registry/node-ca-mq9n4"
Sep 14 09:43:32.282856 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.281993    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkm5l\" (UniqueName: \"kubernetes.io/projected/c41422ae-06c2-492c-9dbf-82e10222589a-kube-api-access-xkm5l\") pod \"network-check-target-8zcts\" (UID: \"c41422ae-06c2-492c-9dbf-82e10222589a\") " pod="openshift-network-diagnostics/network-check-target-8zcts"
Sep 14 09:43:32.282856 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.282007    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dba3fdc7-1cc0-451e-9359-f597a2fb65a3-host-cni-bin\") pod \"ovnkube-node-hdclt\" (UID: \"dba3fdc7-1cc0-451e-9359-f597a2fb65a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdclt"
Sep 14 09:43:32.282856 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.282080    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/da9fcef7-654c-4140-bb22-abd2ece6e0c1-etc-systemd\") pod \"tuned-kb4pz\" (UID: \"da9fcef7-654c-4140-bb22-abd2ece6e0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-kb4pz"
Sep 14 09:43:32.282856 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.282110    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqczc\" (UniqueName: \"kubernetes.io/projected/da9fcef7-654c-4140-bb22-abd2ece6e0c1-kube-api-access-tqczc\") pod \"tuned-kb4pz\" (UID: \"da9fcef7-654c-4140-bb22-abd2ece6e0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-kb4pz"
Sep 14 09:43:32.282856 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.282139    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/da9fcef7-654c-4140-bb22-abd2ece6e0c1-etc-systemd\") pod \"tuned-kb4pz\" (UID: \"da9fcef7-654c-4140-bb22-abd2ece6e0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-kb4pz"
Sep 14 09:43:32.282856 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.282140    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dba3fdc7-1cc0-451e-9359-f597a2fb65a3-run-systemd\") pod \"ovnkube-node-hdclt\" (UID: \"dba3fdc7-1cc0-451e-9359-f597a2fb65a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdclt"
Sep 14 09:43:32.282856 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.282171    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dba3fdc7-1cc0-451e-9359-f597a2fb65a3-host-run-ovn-kubernetes\") pod \"ovnkube-node-hdclt\" (UID: \"dba3fdc7-1cc0-451e-9359-f597a2fb65a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdclt"
Sep 14 09:43:32.282856 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.282174    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dba3fdc7-1cc0-451e-9359-f597a2fb65a3-run-systemd\") pod \"ovnkube-node-hdclt\" (UID: \"dba3fdc7-1cc0-451e-9359-f597a2fb65a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdclt"
Sep 14 09:43:32.282856 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.282203    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dba3fdc7-1cc0-451e-9359-f597a2fb65a3-host-run-ovn-kubernetes\") pod \"ovnkube-node-hdclt\" (UID: \"dba3fdc7-1cc0-451e-9359-f597a2fb65a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdclt"
Sep 14 09:43:32.282856 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.282223    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scdrt\" (UniqueName: \"kubernetes.io/projected/2c7467c2-fbc6-4195-b321-acc1ef713a13-kube-api-access-scdrt\") pod \"iptables-alerter-cgxf2\" (UID: \"2c7467c2-fbc6-4195-b321-acc1ef713a13\") " pod="openshift-network-operator/iptables-alerter-cgxf2"
Sep 14 09:43:32.282856 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.282242    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1c80666a-ce28-47ef-b790-be0563101bb4-host-var-lib-cni-bin\") pod \"multus-tcvsm\" (UID: \"1c80666a-ce28-47ef-b790-be0563101bb4\") " pod="openshift-multus/multus-tcvsm"
Sep 14 09:43:32.282856 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.282258    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d956c018-b9cb-4c19-8f08-16652611d8c3-metrics-certs\") pod \"network-metrics-daemon-6l89k\" (UID: \"d956c018-b9cb-4c19-8f08-16652611d8c3\") " pod="openshift-multus/network-metrics-daemon-6l89k"
Sep 14 09:43:32.282856 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.282298    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dba3fdc7-1cc0-451e-9359-f597a2fb65a3-env-overrides\") pod \"ovnkube-node-hdclt\" (UID: \"dba3fdc7-1cc0-451e-9359-f597a2fb65a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdclt"
Sep 14 09:43:32.282856 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.282354    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0a7d1077-b7fb-479d-990d-e35e6e7611bd-etc-selinux\") pod \"aws-ebs-csi-driver-node-fjwpq\" (UID: \"0a7d1077-b7fb-479d-990d-e35e6e7611bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjwpq"
Sep 14 09:43:32.282856 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.282420    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1c80666a-ce28-47ef-b790-be0563101bb4-hostroot\") pod \"multus-tcvsm\" (UID: \"1c80666a-ce28-47ef-b790-be0563101bb4\") " pod="openshift-multus/multus-tcvsm"
Sep 14 09:43:32.282856 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.282462    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr2dj\" (UniqueName: \"kubernetes.io/projected/20d9618f-664c-4cae-bcc2-4b00ba45acc6-kube-api-access-vr2dj\") pod \"multus-additional-cni-plugins-7tm8l\" (UID: \"20d9618f-664c-4cae-bcc2-4b00ba45acc6\") " pod="openshift-multus/multus-additional-cni-plugins-7tm8l"
Sep 14 09:43:32.283455 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.282525    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2c7467c2-fbc6-4195-b321-acc1ef713a13-host-slash\") pod \"iptables-alerter-cgxf2\" (UID: \"2c7467c2-fbc6-4195-b321-acc1ef713a13\") " pod="openshift-network-operator/iptables-alerter-cgxf2"
Sep 14 09:43:32.283455 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.282566    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/da9fcef7-654c-4140-bb22-abd2ece6e0c1-etc-sysconfig\") pod \"tuned-kb4pz\" (UID: \"da9fcef7-654c-4140-bb22-abd2ece6e0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-kb4pz"
Sep 14 09:43:32.283455 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.282591    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/da9fcef7-654c-4140-bb22-abd2ece6e0c1-host\") pod \"tuned-kb4pz\" (UID: \"da9fcef7-654c-4140-bb22-abd2ece6e0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-kb4pz"
Sep 14 09:43:32.283455 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.282615    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/66c47172-17e5-4985-84a8-987c216dd4ec-agent-certs\") pod \"konnectivity-agent-cskmj\" (UID: \"66c47172-17e5-4985-84a8-987c216dd4ec\") " pod="kube-system/konnectivity-agent-cskmj"
Sep 14 09:43:32.283455 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.282624    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/da9fcef7-654c-4140-bb22-abd2ece6e0c1-etc-sysconfig\") pod \"tuned-kb4pz\" (UID: \"da9fcef7-654c-4140-bb22-abd2ece6e0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-kb4pz"
Sep 14 09:43:32.283455 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.282641    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/20d9618f-664c-4cae-bcc2-4b00ba45acc6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7tm8l\" (UID: \"20d9618f-664c-4cae-bcc2-4b00ba45acc6\") " pod="openshift-multus/multus-additional-cni-plugins-7tm8l"
Sep 14 09:43:32.283455 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.282665    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/97284aa2-9a80-495f-8b99-2ff7cec913da-dbus\") pod \"global-pull-secret-syncer-krg4v\" (UID: \"97284aa2-9a80-495f-8b99-2ff7cec913da\") " pod="kube-system/global-pull-secret-syncer-krg4v"
Sep 14 09:43:32.283455 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.282679    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/da9fcef7-654c-4140-bb22-abd2ece6e0c1-host\") pod \"tuned-kb4pz\" (UID: \"da9fcef7-654c-4140-bb22-abd2ece6e0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-kb4pz"
Sep 14 09:43:32.283455 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.282703    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97284aa2-9a80-495f-8b99-2ff7cec913da-original-pull-secret\") pod \"global-pull-secret-syncer-krg4v\" (UID: \"97284aa2-9a80-495f-8b99-2ff7cec913da\") " pod="kube-system/global-pull-secret-syncer-krg4v"
Sep 14 09:43:32.283455 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.282769    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dba3fdc7-1cc0-451e-9359-f597a2fb65a3-env-overrides\") pod \"ovnkube-node-hdclt\" (UID: \"dba3fdc7-1cc0-451e-9359-f597a2fb65a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdclt"
Sep 14 09:43:32.283455 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.282773    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5s6m\" (UniqueName: \"kubernetes.io/projected/07d5e1df-7d96-401d-b606-245e4b1f3ae5-kube-api-access-x5s6m\") pod \"node-ca-mq9n4\" (UID: \"07d5e1df-7d96-401d-b606-245e4b1f3ae5\") " pod="openshift-image-registry/node-ca-mq9n4"
Sep 14 09:43:32.283455 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.282820    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dba3fdc7-1cc0-451e-9359-f597a2fb65a3-ovnkube-script-lib\") pod \"ovnkube-node-hdclt\" (UID: \"dba3fdc7-1cc0-451e-9359-f597a2fb65a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdclt"
Sep 14 09:43:32.283455 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.282832    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/da9fcef7-654c-4140-bb22-abd2ece6e0c1-etc-sysctl-d\") pod \"tuned-kb4pz\" (UID: \"da9fcef7-654c-4140-bb22-abd2ece6e0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-kb4pz"
Sep 14 09:43:32.283455 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.282860    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dba3fdc7-1cc0-451e-9359-f597a2fb65a3-etc-openvswitch\") pod \"ovnkube-node-hdclt\" (UID: \"dba3fdc7-1cc0-451e-9359-f597a2fb65a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdclt"
Sep 14 09:43:32.283455 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.282881    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mr4p\" (UniqueName: \"kubernetes.io/projected/0a7d1077-b7fb-479d-990d-e35e6e7611bd-kube-api-access-6mr4p\") pod \"aws-ebs-csi-driver-node-fjwpq\" (UID: \"0a7d1077-b7fb-479d-990d-e35e6e7611bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjwpq"
Sep 14 09:43:32.283455 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.282905    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/20d9618f-664c-4cae-bcc2-4b00ba45acc6-cnibin\") pod \"multus-additional-cni-plugins-7tm8l\" (UID: \"20d9618f-664c-4cae-bcc2-4b00ba45acc6\") " pod="openshift-multus/multus-additional-cni-plugins-7tm8l"
Sep 14 09:43:32.283455 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.282929    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/20d9618f-664c-4cae-bcc2-4b00ba45acc6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7tm8l\" (UID: \"20d9618f-664c-4cae-bcc2-4b00ba45acc6\") " pod="openshift-multus/multus-additional-cni-plugins-7tm8l"
Sep 14 09:43:32.284156 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.282945    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/da9fcef7-654c-4140-bb22-abd2ece6e0c1-etc-sysctl-d\") pod \"tuned-kb4pz\" (UID: \"da9fcef7-654c-4140-bb22-abd2ece6e0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-kb4pz"
Sep 14 09:43:32.284156 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.282952    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dba3fdc7-1cc0-451e-9359-f597a2fb65a3-etc-openvswitch\") pod \"ovnkube-node-hdclt\" (UID: \"dba3fdc7-1cc0-451e-9359-f597a2fb65a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdclt"
Sep 14 09:43:32.284156 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.282958    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/da9fcef7-654c-4140-bb22-abd2ece6e0c1-etc-kubernetes\") pod \"tuned-kb4pz\" (UID: \"da9fcef7-654c-4140-bb22-abd2ece6e0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-kb4pz"
Sep 14 09:43:32.284156 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.282988    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn7bs\" (UniqueName: \"kubernetes.io/projected/dba3fdc7-1cc0-451e-9359-f597a2fb65a3-kube-api-access-vn7bs\") pod \"ovnkube-node-hdclt\" (UID: \"dba3fdc7-1cc0-451e-9359-f597a2fb65a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdclt"
Sep 14 09:43:32.284156 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.283017    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c80666a-ce28-47ef-b790-be0563101bb4-etc-kubernetes\") pod \"multus-tcvsm\" (UID: \"1c80666a-ce28-47ef-b790-be0563101bb4\") " pod="openshift-multus/multus-tcvsm"
Sep 14 09:43:32.284156 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.283047    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/da9fcef7-654c-4140-bb22-abd2ece6e0c1-etc-kubernetes\") pod \"tuned-kb4pz\" (UID: \"da9fcef7-654c-4140-bb22-abd2ece6e0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-kb4pz"
Sep 14 09:43:32.284156 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.283092    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/20d9618f-664c-4cae-bcc2-4b00ba45acc6-system-cni-dir\") pod \"multus-additional-cni-plugins-7tm8l\" (UID: \"20d9618f-664c-4cae-bcc2-4b00ba45acc6\") " pod="openshift-multus/multus-additional-cni-plugins-7tm8l"
Sep 14 09:43:32.284156 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.283180    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/da9fcef7-654c-4140-bb22-abd2ece6e0c1-sys\") pod \"tuned-kb4pz\" (UID: \"da9fcef7-654c-4140-bb22-abd2ece6e0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-kb4pz"
Sep 14 09:43:32.284156 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.283214    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dba3fdc7-1cc0-451e-9359-f597a2fb65a3-host-kubelet\") pod \"ovnkube-node-hdclt\" (UID: \"dba3fdc7-1cc0-451e-9359-f597a2fb65a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdclt"
Sep 14 09:43:32.284156 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.283240    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0a7d1077-b7fb-479d-990d-e35e6e7611bd-device-dir\") pod \"aws-ebs-csi-driver-node-fjwpq\" (UID: \"0a7d1077-b7fb-479d-990d-e35e6e7611bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjwpq"
Sep 14 09:43:32.284156 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.283270    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dba3fdc7-1cc0-451e-9359-f597a2fb65a3-host-kubelet\") pod \"ovnkube-node-hdclt\" (UID: \"dba3fdc7-1cc0-451e-9359-f597a2fb65a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdclt"
Sep 14 09:43:32.284156 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.283321    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/da9fcef7-654c-4140-bb22-abd2ece6e0c1-sys\") pod \"tuned-kb4pz\" (UID: \"da9fcef7-654c-4140-bb22-abd2ece6e0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-kb4pz"
Sep 14 09:43:32.284156 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.283329    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1c80666a-ce28-47ef-b790-be0563101bb4-multus-cni-dir\") pod \"multus-tcvsm\" (UID: \"1c80666a-ce28-47ef-b790-be0563101bb4\") " pod="openshift-multus/multus-tcvsm"
Sep 14 09:43:32.284156 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.283354    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1c80666a-ce28-47ef-b790-be0563101bb4-multus-daemon-config\") pod \"multus-tcvsm\" (UID: \"1c80666a-ce28-47ef-b790-be0563101bb4\") " pod="openshift-multus/multus-tcvsm"
Sep 14 09:43:32.284156 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.283376    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1c80666a-ce28-47ef-b790-be0563101bb4-host-run-multus-certs\") pod \"multus-tcvsm\" (UID: \"1c80666a-ce28-47ef-b790-be0563101bb4\") " pod="openshift-multus/multus-tcvsm"
Sep 14 09:43:32.284156 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.283399    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv9s8\" (UniqueName: \"kubernetes.io/projected/d956c018-b9cb-4c19-8f08-16652611d8c3-kube-api-access-bv9s8\") pod \"network-metrics-daemon-6l89k\" (UID: \"d956c018-b9cb-4c19-8f08-16652611d8c3\") " pod="openshift-multus/network-metrics-daemon-6l89k"
Sep 14 09:43:32.284156 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.283430    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/da9fcef7-654c-4140-bb22-abd2ece6e0c1-run\") pod \"tuned-kb4pz\" (UID: \"da9fcef7-654c-4140-bb22-abd2ece6e0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-kb4pz"
Sep 14 09:43:32.284724 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.283454    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dba3fdc7-1cc0-451e-9359-f597a2fb65a3-host-slash\") pod \"ovnkube-node-hdclt\" (UID: \"dba3fdc7-1cc0-451e-9359-f597a2fb65a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdclt"
Sep 14 09:43:32.284724 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.283480    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0a7d1077-b7fb-479d-990d-e35e6e7611bd-sys-fs\") pod \"aws-ebs-csi-driver-node-fjwpq\" (UID: \"0a7d1077-b7fb-479d-990d-e35e6e7611bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjwpq"
Sep 14 09:43:32.284724 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.283504    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6dv9\" (UniqueName: \"kubernetes.io/projected/1c80666a-ce28-47ef-b790-be0563101bb4-kube-api-access-w6dv9\") pod \"multus-tcvsm\" (UID: \"1c80666a-ce28-47ef-b790-be0563101bb4\") " pod="openshift-multus/multus-tcvsm"
Sep 14 09:43:32.284724 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.283511    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dba3fdc7-1cc0-451e-9359-f597a2fb65a3-host-slash\") pod \"ovnkube-node-hdclt\" (UID: \"dba3fdc7-1cc0-451e-9359-f597a2fb65a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdclt"
Sep 14 09:43:32.284724 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.283529    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/20d9618f-664c-4cae-bcc2-4b00ba45acc6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7tm8l\" (UID: \"20d9618f-664c-4cae-bcc2-4b00ba45acc6\") " pod="openshift-multus/multus-additional-cni-plugins-7tm8l"
Sep 14 09:43:32.284724 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.283539    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/da9fcef7-654c-4140-bb22-abd2ece6e0c1-run\") pod \"tuned-kb4pz\" (UID: \"da9fcef7-654c-4140-bb22-abd2ece6e0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-kb4pz"
Sep 14 09:43:32.284724 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.283555    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/07d5e1df-7d96-401d-b606-245e4b1f3ae5-serviceca\") pod \"node-ca-mq9n4\" (UID: \"07d5e1df-7d96-401d-b606-245e4b1f3ae5\") " pod="openshift-image-registry/node-ca-mq9n4"
Sep 14 09:43:32.284724 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.283577    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/da9fcef7-654c-4140-bb22-abd2ece6e0c1-etc-tuned\") pod \"tuned-kb4pz\" (UID: \"da9fcef7-654c-4140-bb22-abd2ece6e0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-kb4pz"
Sep 14 09:43:32.284724 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.283597    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dba3fdc7-1cc0-451e-9359-f597a2fb65a3-ovn-node-metrics-cert\") pod \"ovnkube-node-hdclt\" (UID: \"dba3fdc7-1cc0-451e-9359-f597a2fb65a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdclt"
Sep 14 09:43:32.284724 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.283621    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2c7467c2-fbc6-4195-b321-acc1ef713a13-iptables-alerter-script\") pod \"iptables-alerter-cgxf2\" (UID: \"2c7467c2-fbc6-4195-b321-acc1ef713a13\") " pod="openshift-network-operator/iptables-alerter-cgxf2"
Sep 14 09:43:32.284724 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.283646    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0a7d1077-b7fb-479d-990d-e35e6e7611bd-socket-dir\") pod \"aws-ebs-csi-driver-node-fjwpq\" (UID: \"0a7d1077-b7fb-479d-990d-e35e6e7611bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjwpq"
Sep 14 09:43:32.284724 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.283679    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/da9fcef7-654c-4140-bb22-abd2ece6e0c1-tmp\") pod \"tuned-kb4pz\" (UID: \"da9fcef7-654c-4140-bb22-abd2ece6e0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-kb4pz"
Sep 14 09:43:32.284724 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.283747    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dba3fdc7-1cc0-451e-9359-f597a2fb65a3-host-cni-netd\") pod \"ovnkube-node-hdclt\" (UID: \"dba3fdc7-1cc0-451e-9359-f597a2fb65a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdclt"
Sep 14 09:43:32.284724 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.283813    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/da4b31da-fcbb-4132-bc49-1ecd91bb65bb-hosts-file\") pod \"node-resolver-f5jts\" (UID: \"da4b31da-fcbb-4132-bc49-1ecd91bb65bb\") " pod="openshift-dns/node-resolver-f5jts"
Sep 14 09:43:32.284724 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.283828    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dba3fdc7-1cc0-451e-9359-f597a2fb65a3-host-cni-netd\") pod \"ovnkube-node-hdclt\" (UID: \"dba3fdc7-1cc0-451e-9359-f597a2fb65a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdclt"
Sep 14 09:43:32.284724 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.283858    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da4b31da-fcbb-4132-bc49-1ecd91bb65bb-tmp-dir\") pod \"node-resolver-f5jts\" (UID: \"da4b31da-fcbb-4132-bc49-1ecd91bb65bb\") " pod="openshift-dns/node-resolver-f5jts"
Sep 14 09:43:32.284724 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.283942    2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 14 09:43:32.285324 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.284031    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/07d5e1df-7d96-401d-b606-245e4b1f3ae5-serviceca\") pod \"node-ca-mq9n4\" (UID: \"07d5e1df-7d96-401d-b606-245e4b1f3ae5\") " pod="openshift-image-registry/node-ca-mq9n4"
Sep 14 09:43:32.287378 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.287347    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/da9fcef7-654c-4140-bb22-abd2ece6e0c1-etc-tuned\") pod \"tuned-kb4pz\" (UID: \"da9fcef7-654c-4140-bb22-abd2ece6e0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-kb4pz"
Sep 14 09:43:32.287527 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.287403    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/da9fcef7-654c-4140-bb22-abd2ece6e0c1-tmp\") pod \"tuned-kb4pz\" (UID: \"da9fcef7-654c-4140-bb22-abd2ece6e0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-kb4pz"
Sep 14 09:43:32.287527 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.287491    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dba3fdc7-1cc0-451e-9359-f597a2fb65a3-ovn-node-metrics-cert\") pod \"ovnkube-node-hdclt\" (UID: \"dba3fdc7-1cc0-451e-9359-f597a2fb65a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdclt"
Sep 14 09:43:32.292866 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.292827    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqczc\" (UniqueName: \"kubernetes.io/projected/da9fcef7-654c-4140-bb22-abd2ece6e0c1-kube-api-access-tqczc\") pod \"tuned-kb4pz\" (UID: \"da9fcef7-654c-4140-bb22-abd2ece6e0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-kb4pz"
Sep 14 09:43:32.293006 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.292909    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5s6m\" (UniqueName: \"kubernetes.io/projected/07d5e1df-7d96-401d-b606-245e4b1f3ae5-kube-api-access-x5s6m\") pod \"node-ca-mq9n4\" (UID: \"07d5e1df-7d96-401d-b606-245e4b1f3ae5\") " pod="openshift-image-registry/node-ca-mq9n4"
Sep 14 09:43:32.293125 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.293104    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn7bs\" (UniqueName: \"kubernetes.io/projected/dba3fdc7-1cc0-451e-9359-f597a2fb65a3-kube-api-access-vn7bs\") pod \"ovnkube-node-hdclt\" (UID: \"dba3fdc7-1cc0-451e-9359-f597a2fb65a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-hdclt"
Sep 14 09:43:32.321202 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.321144    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-4.ec2.internal" event={"ID":"812604c7944c05f98e0873536f91322d","Type":"ContainerStarted","Data":"5ab647349c219fe21c2cfa151e21dd50928af9db88c3e1dd33136b20b7e3b854"}
Sep 14 09:43:32.322932 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.322901    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal" event={"ID":"53b749d177a0ad18e520172c7d6ddac0","Type":"ContainerStarted","Data":"cc7858d9f12b827f6ae407d2b3995e37d9fdcbf7508aa91878e50c6acc9f43e4"}
Sep 14 09:43:32.384390 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.384344    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1c80666a-ce28-47ef-b790-be0563101bb4-multus-socket-dir-parent\") pod \"multus-tcvsm\" (UID: \"1c80666a-ce28-47ef-b790-be0563101bb4\") " pod="openshift-multus/multus-tcvsm"
Sep 14 09:43:32.384580 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.384398    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1c80666a-ce28-47ef-b790-be0563101bb4-host-run-netns\") pod \"multus-tcvsm\" (UID: \"1c80666a-ce28-47ef-b790-be0563101bb4\") " pod="openshift-multus/multus-tcvsm"
Sep 14 09:43:32.384580 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.384429    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xtjs\" (UniqueName: \"kubernetes.io/projected/da4b31da-fcbb-4132-bc49-1ecd91bb65bb-kube-api-access-9xtjs\") pod \"node-resolver-f5jts\" (UID: \"da4b31da-fcbb-4132-bc49-1ecd91bb65bb\") " pod="openshift-dns/node-resolver-f5jts"
Sep 14 09:43:32.384580 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.384456    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0a7d1077-b7fb-479d-990d-e35e6e7611bd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fjwpq\" (UID: \"0a7d1077-b7fb-479d-990d-e35e6e7611bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjwpq"
Sep 14 09:43:32.384580 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.384483    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkm5l\" (UniqueName: \"kubernetes.io/projected/c41422ae-06c2-492c-9dbf-82e10222589a-kube-api-access-xkm5l\") pod \"network-check-target-8zcts\" (UID: \"c41422ae-06c2-492c-9dbf-82e10222589a\") " pod="openshift-network-diagnostics/network-check-target-8zcts"
Sep 14 09:43:32.384580 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.384489    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1c80666a-ce28-47ef-b790-be0563101bb4-multus-socket-dir-parent\") pod \"multus-tcvsm\" (UID: \"1c80666a-ce28-47ef-b790-be0563101bb4\") " pod="openshift-multus/multus-tcvsm"
Sep 14 09:43:32.384580 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.384513    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scdrt\" (UniqueName: \"kubernetes.io/projected/2c7467c2-fbc6-4195-b321-acc1ef713a13-kube-api-access-scdrt\") pod \"iptables-alerter-cgxf2\" (UID: \"2c7467c2-fbc6-4195-b321-acc1ef713a13\") " pod="openshift-network-operator/iptables-alerter-cgxf2"
Sep 14 09:43:32.384580 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.384561    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1c80666a-ce28-47ef-b790-be0563101bb4-host-var-lib-cni-bin\") pod \"multus-tcvsm\" (UID: \"1c80666a-ce28-47ef-b790-be0563101bb4\") " pod="openshift-multus/multus-tcvsm"
Sep 14 09:43:32.384580 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.384579    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1c80666a-ce28-47ef-b790-be0563101bb4-host-run-netns\") pod \"multus-tcvsm\" (UID: \"1c80666a-ce28-47ef-b790-be0563101bb4\") " pod="openshift-multus/multus-tcvsm"
Sep 14 09:43:32.385010 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.384594    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d956c018-b9cb-4c19-8f08-16652611d8c3-metrics-certs\") pod \"network-metrics-daemon-6l89k\" (UID: \"d956c018-b9cb-4c19-8f08-16652611d8c3\") " pod="openshift-multus/network-metrics-daemon-6l89k"
Sep 14 09:43:32.385010 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.384621    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0a7d1077-b7fb-479d-990d-e35e6e7611bd-etc-selinux\") pod \"aws-ebs-csi-driver-node-fjwpq\" (UID: \"0a7d1077-b7fb-479d-990d-e35e6e7611bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjwpq"
Sep 14 09:43:32.385010 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.384646    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1c80666a-ce28-47ef-b790-be0563101bb4-hostroot\") pod \"multus-tcvsm\" (UID: \"1c80666a-ce28-47ef-b790-be0563101bb4\") " pod="openshift-multus/multus-tcvsm"
Sep 14 09:43:32.385010 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.384673    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr2dj\" (UniqueName: \"kubernetes.io/projected/20d9618f-664c-4cae-bcc2-4b00ba45acc6-kube-api-access-vr2dj\") pod \"multus-additional-cni-plugins-7tm8l\" (UID: \"20d9618f-664c-4cae-bcc2-4b00ba45acc6\") " pod="openshift-multus/multus-additional-cni-plugins-7tm8l"
Sep 14 09:43:32.385010 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.384721    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2c7467c2-fbc6-4195-b321-acc1ef713a13-host-slash\") pod \"iptables-alerter-cgxf2\" (UID: \"2c7467c2-fbc6-4195-b321-acc1ef713a13\") " pod="openshift-network-operator/iptables-alerter-cgxf2"
Sep 14 09:43:32.385010 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.384747    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/66c47172-17e5-4985-84a8-987c216dd4ec-agent-certs\") pod \"konnectivity-agent-cskmj\" (UID: \"66c47172-17e5-4985-84a8-987c216dd4ec\") " pod="kube-system/konnectivity-agent-cskmj"
Sep 14 09:43:32.385010 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.384773    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/20d9618f-664c-4cae-bcc2-4b00ba45acc6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7tm8l\" (UID: \"20d9618f-664c-4cae-bcc2-4b00ba45acc6\") " pod="openshift-multus/multus-additional-cni-plugins-7tm8l"
Sep 14 09:43:32.385010 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.384783    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0a7d1077-b7fb-479d-990d-e35e6e7611bd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fjwpq\" (UID: \"0a7d1077-b7fb-479d-990d-e35e6e7611bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjwpq"
Sep 14 09:43:32.385010 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.384795    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/97284aa2-9a80-495f-8b99-2ff7cec913da-dbus\") pod \"global-pull-secret-syncer-krg4v\" (UID: \"97284aa2-9a80-495f-8b99-2ff7cec913da\") " pod="kube-system/global-pull-secret-syncer-krg4v"
Sep 14 09:43:32.385010 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.384827    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97284aa2-9a80-495f-8b99-2ff7cec913da-original-pull-secret\") pod \"global-pull-secret-syncer-krg4v\" (UID: \"97284aa2-9a80-495f-8b99-2ff7cec913da\") " pod="kube-system/global-pull-secret-syncer-krg4v"
Sep 14 09:43:32.385010 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.384842    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0a7d1077-b7fb-479d-990d-e35e6e7611bd-etc-selinux\") pod \"aws-ebs-csi-driver-node-fjwpq\" (UID: \"0a7d1077-b7fb-479d-990d-e35e6e7611bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjwpq"
Sep 14 09:43:32.385010 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.384854    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mr4p\" (UniqueName: \"kubernetes.io/projected/0a7d1077-b7fb-479d-990d-e35e6e7611bd-kube-api-access-6mr4p\") pod \"aws-ebs-csi-driver-node-fjwpq\" (UID: \"0a7d1077-b7fb-479d-990d-e35e6e7611bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjwpq"
Sep 14 09:43:32.385010 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.384871    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2c7467c2-fbc6-4195-b321-acc1ef713a13-host-slash\") pod \"iptables-alerter-cgxf2\" (UID: \"2c7467c2-fbc6-4195-b321-acc1ef713a13\") " pod="openshift-network-operator/iptables-alerter-cgxf2"
Sep 14 09:43:32.385010 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.384885    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1c80666a-ce28-47ef-b790-be0563101bb4-hostroot\") pod \"multus-tcvsm\" (UID: \"1c80666a-ce28-47ef-b790-be0563101bb4\") " pod="openshift-multus/multus-tcvsm"
Sep 14 09:43:32.385010 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.384878    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/20d9618f-664c-4cae-bcc2-4b00ba45acc6-cnibin\") pod \"multus-additional-cni-plugins-7tm8l\" (UID: \"20d9618f-664c-4cae-bcc2-4b00ba45acc6\") " pod="openshift-multus/multus-additional-cni-plugins-7tm8l"
Sep 14 09:43:32.385010 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.384906    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/20d9618f-664c-4cae-bcc2-4b00ba45acc6-cnibin\") pod \"multus-additional-cni-plugins-7tm8l\" (UID: \"20d9618f-664c-4cae-bcc2-4b00ba45acc6\") " pod="openshift-multus/multus-additional-cni-plugins-7tm8l"
Sep 14 09:43:32.385010 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.384926    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/97284aa2-9a80-495f-8b99-2ff7cec913da-dbus\") pod \"global-pull-secret-syncer-krg4v\" (UID: \"97284aa2-9a80-495f-8b99-2ff7cec913da\") " pod="kube-system/global-pull-secret-syncer-krg4v"
Sep 14 09:43:32.385852 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.384937    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/20d9618f-664c-4cae-bcc2-4b00ba45acc6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7tm8l\" (UID: \"20d9618f-664c-4cae-bcc2-4b00ba45acc6\") " pod="openshift-multus/multus-additional-cni-plugins-7tm8l"
Sep 14 09:43:32.385852 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:32.384958    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 14 09:43:32.385852 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:32.384971    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 09:43:32.385852 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.385003    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c80666a-ce28-47ef-b790-be0563101bb4-etc-kubernetes\") pod \"multus-tcvsm\" (UID: \"1c80666a-ce28-47ef-b790-be0563101bb4\") " pod="openshift-multus/multus-tcvsm"
Sep 14 09:43:32.385852 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.384969    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c80666a-ce28-47ef-b790-be0563101bb4-etc-kubernetes\") pod \"multus-tcvsm\" (UID: \"1c80666a-ce28-47ef-b790-be0563101bb4\") " pod="openshift-multus/multus-tcvsm"
Sep 14 09:43:32.385852 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.384882    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1c80666a-ce28-47ef-b790-be0563101bb4-host-var-lib-cni-bin\") pod \"multus-tcvsm\" (UID: \"1c80666a-ce28-47ef-b790-be0563101bb4\") " pod="openshift-multus/multus-tcvsm"
Sep 14 09:43:32.385852 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:32.385046    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97284aa2-9a80-495f-8b99-2ff7cec913da-original-pull-secret podName:97284aa2-9a80-495f-8b99-2ff7cec913da nodeName:}" failed. No retries permitted until 2026-09-14 09:43:32.885007168 +0000 UTC m=+3.083992255 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/97284aa2-9a80-495f-8b99-2ff7cec913da-original-pull-secret") pod "global-pull-secret-syncer-krg4v" (UID: "97284aa2-9a80-495f-8b99-2ff7cec913da") : object "kube-system"/"original-pull-secret" not registered
Sep 14 09:43:32.385852 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:32.385162    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d956c018-b9cb-4c19-8f08-16652611d8c3-metrics-certs podName:d956c018-b9cb-4c19-8f08-16652611d8c3 nodeName:}" failed. No retries permitted until 2026-09-14 09:43:32.885153538 +0000 UTC m=+3.084138610 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d956c018-b9cb-4c19-8f08-16652611d8c3-metrics-certs") pod "network-metrics-daemon-6l89k" (UID: "d956c018-b9cb-4c19-8f08-16652611d8c3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 09:43:32.385852 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.385184    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/20d9618f-664c-4cae-bcc2-4b00ba45acc6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7tm8l\" (UID: \"20d9618f-664c-4cae-bcc2-4b00ba45acc6\") " pod="openshift-multus/multus-additional-cni-plugins-7tm8l"
Sep 14 09:43:32.385852 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.385190    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/20d9618f-664c-4cae-bcc2-4b00ba45acc6-system-cni-dir\") pod \"multus-additional-cni-plugins-7tm8l\" (UID: \"20d9618f-664c-4cae-bcc2-4b00ba45acc6\") " pod="openshift-multus/multus-additional-cni-plugins-7tm8l"
Sep 14 09:43:32.385852 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.385220    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0a7d1077-b7fb-479d-990d-e35e6e7611bd-device-dir\") pod \"aws-ebs-csi-driver-node-fjwpq\" (UID: \"0a7d1077-b7fb-479d-990d-e35e6e7611bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjwpq"
Sep 14 09:43:32.385852 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.385226    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/20d9618f-664c-4cae-bcc2-4b00ba45acc6-system-cni-dir\") pod \"multus-additional-cni-plugins-7tm8l\" (UID: \"20d9618f-664c-4cae-bcc2-4b00ba45acc6\") " pod="openshift-multus/multus-additional-cni-plugins-7tm8l"
Sep 14 09:43:32.385852 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.385251    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1c80666a-ce28-47ef-b790-be0563101bb4-multus-cni-dir\") pod \"multus-tcvsm\" (UID: \"1c80666a-ce28-47ef-b790-be0563101bb4\") " pod="openshift-multus/multus-tcvsm"
Sep 14 09:43:32.385852 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.385274    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1c80666a-ce28-47ef-b790-be0563101bb4-multus-daemon-config\") pod \"multus-tcvsm\" (UID: \"1c80666a-ce28-47ef-b790-be0563101bb4\") " pod="openshift-multus/multus-tcvsm"
Sep 14 09:43:32.385852 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.385299    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1c80666a-ce28-47ef-b790-be0563101bb4-host-run-multus-certs\") pod \"multus-tcvsm\" (UID: \"1c80666a-ce28-47ef-b790-be0563101bb4\") " pod="openshift-multus/multus-tcvsm"
Sep 14 09:43:32.385852 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.385316    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1c80666a-ce28-47ef-b790-be0563101bb4-multus-cni-dir\") pod \"multus-tcvsm\" (UID: \"1c80666a-ce28-47ef-b790-be0563101bb4\") " pod="openshift-multus/multus-tcvsm"
Sep 14 09:43:32.385852 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.385276    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0a7d1077-b7fb-479d-990d-e35e6e7611bd-device-dir\") pod \"aws-ebs-csi-driver-node-fjwpq\" (UID: \"0a7d1077-b7fb-479d-990d-e35e6e7611bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjwpq"
Sep 14 09:43:32.386601 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.385344    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1c80666a-ce28-47ef-b790-be0563101bb4-host-run-multus-certs\") pod \"multus-tcvsm\" (UID: \"1c80666a-ce28-47ef-b790-be0563101bb4\") " pod="openshift-multus/multus-tcvsm"
Sep 14 09:43:32.386601 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.385380    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv9s8\" (UniqueName: \"kubernetes.io/projected/d956c018-b9cb-4c19-8f08-16652611d8c3-kube-api-access-bv9s8\") pod \"network-metrics-daemon-6l89k\" (UID: \"d956c018-b9cb-4c19-8f08-16652611d8c3\") " pod="openshift-multus/network-metrics-daemon-6l89k"
Sep 14 09:43:32.386601 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.385511    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0a7d1077-b7fb-479d-990d-e35e6e7611bd-sys-fs\") pod \"aws-ebs-csi-driver-node-fjwpq\" (UID: \"0a7d1077-b7fb-479d-990d-e35e6e7611bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjwpq"
Sep 14 09:43:32.386601 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.385542    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6dv9\" (UniqueName: \"kubernetes.io/projected/1c80666a-ce28-47ef-b790-be0563101bb4-kube-api-access-w6dv9\") pod \"multus-tcvsm\" (UID: \"1c80666a-ce28-47ef-b790-be0563101bb4\") " pod="openshift-multus/multus-tcvsm"
Sep 14 09:43:32.386601 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.385570    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/20d9618f-664c-4cae-bcc2-4b00ba45acc6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7tm8l\" (UID: \"20d9618f-664c-4cae-bcc2-4b00ba45acc6\") " pod="openshift-multus/multus-additional-cni-plugins-7tm8l"
Sep 14 09:43:32.386601 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.385570    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/20d9618f-664c-4cae-bcc2-4b00ba45acc6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7tm8l\" (UID: \"20d9618f-664c-4cae-bcc2-4b00ba45acc6\") " pod="openshift-multus/multus-additional-cni-plugins-7tm8l"
Sep 14 09:43:32.386601 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.385618    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0a7d1077-b7fb-479d-990d-e35e6e7611bd-sys-fs\") pod \"aws-ebs-csi-driver-node-fjwpq\" (UID: \"0a7d1077-b7fb-479d-990d-e35e6e7611bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjwpq"
Sep 14 09:43:32.386601 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.385607    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2c7467c2-fbc6-4195-b321-acc1ef713a13-iptables-alerter-script\") pod \"iptables-alerter-cgxf2\" (UID: \"2c7467c2-fbc6-4195-b321-acc1ef713a13\") " pod="openshift-network-operator/iptables-alerter-cgxf2"
Sep 14 09:43:32.386601 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.385651    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0a7d1077-b7fb-479d-990d-e35e6e7611bd-socket-dir\") pod \"aws-ebs-csi-driver-node-fjwpq\" (UID: \"0a7d1077-b7fb-479d-990d-e35e6e7611bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjwpq"
Sep 14 09:43:32.386601 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.385682    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/da4b31da-fcbb-4132-bc49-1ecd91bb65bb-hosts-file\") pod \"node-resolver-f5jts\" (UID: \"da4b31da-fcbb-4132-bc49-1ecd91bb65bb\") " pod="openshift-dns/node-resolver-f5jts"
Sep 14 09:43:32.386601 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.385717    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da4b31da-fcbb-4132-bc49-1ecd91bb65bb-tmp-dir\") pod \"node-resolver-f5jts\" (UID: \"da4b31da-fcbb-4132-bc49-1ecd91bb65bb\") " pod="openshift-dns/node-resolver-f5jts"
Sep 14 09:43:32.386601 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.385747    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/66c47172-17e5-4985-84a8-987c216dd4ec-konnectivity-ca\") pod \"konnectivity-agent-cskmj\" (UID: \"66c47172-17e5-4985-84a8-987c216dd4ec\") " pod="kube-system/konnectivity-agent-cskmj"
Sep 14 09:43:32.386601 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.385772    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1c80666a-ce28-47ef-b790-be0563101bb4-cnibin\") pod \"multus-tcvsm\" (UID: \"1c80666a-ce28-47ef-b790-be0563101bb4\") " pod="openshift-multus/multus-tcvsm"
Sep 14 09:43:32.386601 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.385794    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1c80666a-ce28-47ef-b790-be0563101bb4-cni-binary-copy\") pod \"multus-tcvsm\" (UID: \"1c80666a-ce28-47ef-b790-be0563101bb4\") " pod="openshift-multus/multus-tcvsm"
Sep 14 09:43:32.386601 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.385821    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0a7d1077-b7fb-479d-990d-e35e6e7611bd-registration-dir\") pod \"aws-ebs-csi-driver-node-fjwpq\" (UID: \"0a7d1077-b7fb-479d-990d-e35e6e7611bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjwpq"
Sep 14 09:43:32.386601 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.385834    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0a7d1077-b7fb-479d-990d-e35e6e7611bd-socket-dir\") pod \"aws-ebs-csi-driver-node-fjwpq\" (UID: \"0a7d1077-b7fb-479d-990d-e35e6e7611bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjwpq"
Sep 14 09:43:32.386601 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.385844    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1c80666a-ce28-47ef-b790-be0563101bb4-system-cni-dir\") pod \"multus-tcvsm\" (UID: \"1c80666a-ce28-47ef-b790-be0563101bb4\") " pod="openshift-multus/multus-tcvsm"
Sep 14 09:43:32.387238 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.385892    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1c80666a-ce28-47ef-b790-be0563101bb4-cnibin\") pod \"multus-tcvsm\" (UID: \"1c80666a-ce28-47ef-b790-be0563101bb4\") " pod="openshift-multus/multus-tcvsm"
Sep 14 09:43:32.387238 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.385896    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1c80666a-ce28-47ef-b790-be0563101bb4-system-cni-dir\") pod \"multus-tcvsm\" (UID: \"1c80666a-ce28-47ef-b790-be0563101bb4\") " pod="openshift-multus/multus-tcvsm"
Sep 14 09:43:32.387238 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.385929    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1c80666a-ce28-47ef-b790-be0563101bb4-multus-daemon-config\") pod \"multus-tcvsm\" (UID: \"1c80666a-ce28-47ef-b790-be0563101bb4\") " pod="openshift-multus/multus-tcvsm"
Sep 14 09:43:32.387238 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.385959    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0a7d1077-b7fb-479d-990d-e35e6e7611bd-registration-dir\") pod \"aws-ebs-csi-driver-node-fjwpq\" (UID: \"0a7d1077-b7fb-479d-990d-e35e6e7611bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjwpq"
Sep 14 09:43:32.387238 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.385987    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1c80666a-ce28-47ef-b790-be0563101bb4-host-run-k8s-cni-cncf-io\") pod \"multus-tcvsm\" (UID: \"1c80666a-ce28-47ef-b790-be0563101bb4\") " pod="openshift-multus/multus-tcvsm"
Sep 14 09:43:32.387238 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.385994    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/da4b31da-fcbb-4132-bc49-1ecd91bb65bb-hosts-file\") pod \"node-resolver-f5jts\" (UID: \"da4b31da-fcbb-4132-bc49-1ecd91bb65bb\") " pod="openshift-dns/node-resolver-f5jts"
Sep 14 09:43:32.387238 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.386010    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c80666a-ce28-47ef-b790-be0563101bb4-host-var-lib-kubelet\") pod \"multus-tcvsm\" (UID: \"1c80666a-ce28-47ef-b790-be0563101bb4\") " pod="openshift-multus/multus-tcvsm"
Sep 14 09:43:32.387238 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.386033    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1c80666a-ce28-47ef-b790-be0563101bb4-host-run-k8s-cni-cncf-io\") pod \"multus-tcvsm\" (UID: \"1c80666a-ce28-47ef-b790-be0563101bb4\") " pod="openshift-multus/multus-tcvsm"
Sep 14 09:43:32.387238 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.386037    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1c80666a-ce28-47ef-b790-be0563101bb4-multus-conf-dir\") pod \"multus-tcvsm\" (UID: \"1c80666a-ce28-47ef-b790-be0563101bb4\") " pod="openshift-multus/multus-tcvsm"
Sep 14 09:43:32.387238 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.386059    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/20d9618f-664c-4cae-bcc2-4b00ba45acc6-cni-binary-copy\") pod \"multus-additional-cni-plugins-7tm8l\" (UID: \"20d9618f-664c-4cae-bcc2-4b00ba45acc6\") " pod="openshift-multus/multus-additional-cni-plugins-7tm8l"
Sep 14 09:43:32.387238 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.386068    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c80666a-ce28-47ef-b790-be0563101bb4-host-var-lib-kubelet\") pod \"multus-tcvsm\" (UID: \"1c80666a-ce28-47ef-b790-be0563101bb4\") " pod="openshift-multus/multus-tcvsm"
Sep 14 09:43:32.387238 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.386083    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/97284aa2-9a80-495f-8b99-2ff7cec913da-kubelet-config\") pod \"global-pull-secret-syncer-krg4v\" (UID: \"97284aa2-9a80-495f-8b99-2ff7cec913da\") " pod="kube-system/global-pull-secret-syncer-krg4v"
Sep 14 09:43:32.387238 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.386094    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da4b31da-fcbb-4132-bc49-1ecd91bb65bb-tmp-dir\") pod \"node-resolver-f5jts\" (UID: \"da4b31da-fcbb-4132-bc49-1ecd91bb65bb\") " pod="openshift-dns/node-resolver-f5jts"
Sep 14 09:43:32.387238 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.386104    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1c80666a-ce28-47ef-b790-be0563101bb4-multus-conf-dir\") pod \"multus-tcvsm\" (UID: \"1c80666a-ce28-47ef-b790-be0563101bb4\") " pod="openshift-multus/multus-tcvsm"
Sep 14 09:43:32.387238 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.386112    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1c80666a-ce28-47ef-b790-be0563101bb4-host-var-lib-cni-multus\") pod \"multus-tcvsm\" (UID: \"1c80666a-ce28-47ef-b790-be0563101bb4\") " pod="openshift-multus/multus-tcvsm"
Sep 14 09:43:32.387238 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.386108    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/20d9618f-664c-4cae-bcc2-4b00ba45acc6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7tm8l\" (UID: \"20d9618f-664c-4cae-bcc2-4b00ba45acc6\") " pod="openshift-multus/multus-additional-cni-plugins-7tm8l"
Sep 14 09:43:32.387238 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.386146    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/97284aa2-9a80-495f-8b99-2ff7cec913da-kubelet-config\") pod \"global-pull-secret-syncer-krg4v\" (UID: \"97284aa2-9a80-495f-8b99-2ff7cec913da\") " pod="kube-system/global-pull-secret-syncer-krg4v"
Sep 14 09:43:32.387238 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.386183    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1c80666a-ce28-47ef-b790-be0563101bb4-host-var-lib-cni-multus\") pod \"multus-tcvsm\" (UID: \"1c80666a-ce28-47ef-b790-be0563101bb4\") " pod="openshift-multus/multus-tcvsm"
Sep 14 09:43:32.387900 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.386301    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2c7467c2-fbc6-4195-b321-acc1ef713a13-iptables-alerter-script\") pod \"iptables-alerter-cgxf2\" (UID: \"2c7467c2-fbc6-4195-b321-acc1ef713a13\") " pod="openshift-network-operator/iptables-alerter-cgxf2"
Sep 14 09:43:32.387900 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.386365    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/66c47172-17e5-4985-84a8-987c216dd4ec-konnectivity-ca\") pod \"konnectivity-agent-cskmj\" (UID: \"66c47172-17e5-4985-84a8-987c216dd4ec\") " pod="kube-system/konnectivity-agent-cskmj"
Sep 14 09:43:32.387900 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.386560    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1c80666a-ce28-47ef-b790-be0563101bb4-cni-binary-copy\") pod \"multus-tcvsm\" (UID: \"1c80666a-ce28-47ef-b790-be0563101bb4\") " pod="openshift-multus/multus-tcvsm"
Sep 14 09:43:32.387900 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.386560    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/20d9618f-664c-4cae-bcc2-4b00ba45acc6-cni-binary-copy\") pod \"multus-additional-cni-plugins-7tm8l\" (UID: \"20d9618f-664c-4cae-bcc2-4b00ba45acc6\") " pod="openshift-multus/multus-additional-cni-plugins-7tm8l"
Sep 14 09:43:32.387900 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.387894    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/66c47172-17e5-4985-84a8-987c216dd4ec-agent-certs\") pod \"konnectivity-agent-cskmj\" (UID: \"66c47172-17e5-4985-84a8-987c216dd4ec\") " pod="kube-system/konnectivity-agent-cskmj"
Sep 14 09:43:32.395428 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.395398    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mr4p\" (UniqueName: \"kubernetes.io/projected/0a7d1077-b7fb-479d-990d-e35e6e7611bd-kube-api-access-6mr4p\") pod \"aws-ebs-csi-driver-node-fjwpq\" (UID: \"0a7d1077-b7fb-479d-990d-e35e6e7611bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjwpq"
Sep 14 09:43:32.396502 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:32.396479    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 14 09:43:32.396502 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:32.396503    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 14 09:43:32.396502 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:32.396516    2563 projected.go:196] Error preparing data for projected volume kube-api-access-xkm5l for pod openshift-network-diagnostics/network-check-target-8zcts: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 09:43:32.396786 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:32.396628    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c41422ae-06c2-492c-9dbf-82e10222589a-kube-api-access-xkm5l podName:c41422ae-06c2-492c-9dbf-82e10222589a nodeName:}" failed. No retries permitted until 2026-09-14 09:43:32.896607768 +0000 UTC m=+3.095592844 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-xkm5l" (UniqueName: "kubernetes.io/projected/c41422ae-06c2-492c-9dbf-82e10222589a-kube-api-access-xkm5l") pod "network-check-target-8zcts" (UID: "c41422ae-06c2-492c-9dbf-82e10222589a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 09:43:32.398719 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.398675    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scdrt\" (UniqueName: \"kubernetes.io/projected/2c7467c2-fbc6-4195-b321-acc1ef713a13-kube-api-access-scdrt\") pod \"iptables-alerter-cgxf2\" (UID: \"2c7467c2-fbc6-4195-b321-acc1ef713a13\") " pod="openshift-network-operator/iptables-alerter-cgxf2"
Sep 14 09:43:32.399217 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.399188    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6dv9\" (UniqueName: \"kubernetes.io/projected/1c80666a-ce28-47ef-b790-be0563101bb4-kube-api-access-w6dv9\") pod \"multus-tcvsm\" (UID: \"1c80666a-ce28-47ef-b790-be0563101bb4\") " pod="openshift-multus/multus-tcvsm"
Sep 14 09:43:32.400080 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.400016    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr2dj\" (UniqueName: \"kubernetes.io/projected/20d9618f-664c-4cae-bcc2-4b00ba45acc6-kube-api-access-vr2dj\") pod \"multus-additional-cni-plugins-7tm8l\" (UID: \"20d9618f-664c-4cae-bcc2-4b00ba45acc6\") " pod="openshift-multus/multus-additional-cni-plugins-7tm8l"
Sep 14 09:43:32.400334 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.400310    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv9s8\" (UniqueName: \"kubernetes.io/projected/d956c018-b9cb-4c19-8f08-16652611d8c3-kube-api-access-bv9s8\") pod \"network-metrics-daemon-6l89k\" (UID: \"d956c018-b9cb-4c19-8f08-16652611d8c3\") " pod="openshift-multus/network-metrics-daemon-6l89k"
Sep 14 09:43:32.400423 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:32.400365    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xtjs\" (UniqueName: \"kubernetes.io/projected/da4b31da-fcbb-4132-bc49-1ecd91bb65bb-kube-api-access-9xtjs\") pod \"node-resolver-f5jts\" (UID: \"da4b31da-fcbb-4132-bc49-1ecd91bb65bb\") " pod="openshift-dns/node-resolver-f5jts"
Sep 14 09:43:33.641128 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:33.641083    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mq9n4"
Sep 14 09:43:33.648915 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:33.648883    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kb4pz"
Sep 14 09:43:33.659897 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:33.659867    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hdclt"
Sep 14 09:43:33.666066 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:33.666035    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cskmj"
Sep 14 09:43:33.672314 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:33.672286    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjwpq"
Sep 14 09:43:33.680251 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:33.680217    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tcvsm"
Sep 14 09:43:33.688303 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:33.688266    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7tm8l"
Sep 14 09:43:33.698175 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:33.698140    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cgxf2"
Sep 14 09:43:33.705080 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:33.705049    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-f5jts"
Sep 14 09:43:33.851664 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:33.851581    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 14 09:43:34.048861 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:34.048824    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97284aa2-9a80-495f-8b99-2ff7cec913da-original-pull-secret\") pod \"global-pull-secret-syncer-krg4v\" (UID: \"97284aa2-9a80-495f-8b99-2ff7cec913da\") " pod="kube-system/global-pull-secret-syncer-krg4v"
Sep 14 09:43:34.049103 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:34.049021    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 14 09:43:34.049103 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:34.049035    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d956c018-b9cb-4c19-8f08-16652611d8c3-metrics-certs\") pod \"network-metrics-daemon-6l89k\" (UID: \"d956c018-b9cb-4c19-8f08-16652611d8c3\") " pod="openshift-multus/network-metrics-daemon-6l89k"
Sep 14 09:43:34.049103 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:34.049089    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97284aa2-9a80-495f-8b99-2ff7cec913da-original-pull-secret podName:97284aa2-9a80-495f-8b99-2ff7cec913da nodeName:}" failed. No retries permitted until 2026-09-14 09:43:35.049069697 +0000 UTC m=+4.088744295 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/97284aa2-9a80-495f-8b99-2ff7cec913da-original-pull-secret") pod "global-pull-secret-syncer-krg4v" (UID: "97284aa2-9a80-495f-8b99-2ff7cec913da") : object "kube-system"/"original-pull-secret" not registered
Sep 14 09:43:34.049276 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:34.049138    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 09:43:34.049276 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:34.049194    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d956c018-b9cb-4c19-8f08-16652611d8c3-metrics-certs podName:d956c018-b9cb-4c19-8f08-16652611d8c3 nodeName:}" failed. No retries permitted until 2026-09-14 09:43:35.049176118 +0000 UTC m=+4.088850722 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d956c018-b9cb-4c19-8f08-16652611d8c3-metrics-certs") pod "network-metrics-daemon-6l89k" (UID: "d956c018-b9cb-4c19-8f08-16652611d8c3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 09:43:34.150046 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:34.149935    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkm5l\" (UniqueName: \"kubernetes.io/projected/c41422ae-06c2-492c-9dbf-82e10222589a-kube-api-access-xkm5l\") pod \"network-check-target-8zcts\" (UID: \"c41422ae-06c2-492c-9dbf-82e10222589a\") " pod="openshift-network-diagnostics/network-check-target-8zcts"
Sep 14 09:43:34.150208 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:34.150125    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 14 09:43:34.150208 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:34.150150    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 14 09:43:34.150208 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:34.150164    2563 projected.go:196] Error preparing data for projected volume kube-api-access-xkm5l for pod openshift-network-diagnostics/network-check-target-8zcts: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 09:43:34.150361 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:34.150244    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c41422ae-06c2-492c-9dbf-82e10222589a-kube-api-access-xkm5l podName:c41422ae-06c2-492c-9dbf-82e10222589a nodeName:}" failed. No retries permitted until 2026-09-14 09:43:35.150213073 +0000 UTC m=+4.189887695 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-xkm5l" (UniqueName: "kubernetes.io/projected/c41422ae-06c2-492c-9dbf-82e10222589a-kube-api-access-xkm5l") pod "network-check-target-8zcts" (UID: "c41422ae-06c2-492c-9dbf-82e10222589a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 09:43:34.399911 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:34.399851    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-13 09:38:33 +0000 UTC" deadline="2028-05-15 03:08:31.263693455 +0000 UTC"
Sep 14 09:43:34.399911 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:34.399907    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14609h24m56.863791097s"
Sep 14 09:43:34.475548 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:34.475515    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20d9618f_664c_4cae_bcc2_4b00ba45acc6.slice/crio-baebe2d6cfbc0088752a8c1444d30f7f672dcb114b506837da8fa0a956ccbf4a WatchSource:0}: Error finding container baebe2d6cfbc0088752a8c1444d30f7f672dcb114b506837da8fa0a956ccbf4a: Status 404 returned error can't find the container with id baebe2d6cfbc0088752a8c1444d30f7f672dcb114b506837da8fa0a956ccbf4a
Sep 14 09:43:34.484366 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:34.484334    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7tm8l" event={"ID":"20d9618f-664c-4cae-bcc2-4b00ba45acc6","Type":"ContainerStarted","Data":"baebe2d6cfbc0088752a8c1444d30f7f672dcb114b506837da8fa0a956ccbf4a"}
Sep 14 09:43:34.497748 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:34.497596    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07d5e1df_7d96_401d_b606_245e4b1f3ae5.slice/crio-1faf766f5c121e58bafdfee51438cfe62a6cd72be735aef9753bfdfc7b27ecec WatchSource:0}: Error finding container 1faf766f5c121e58bafdfee51438cfe62a6cd72be735aef9753bfdfc7b27ecec: Status 404 returned error can't find the container with id 1faf766f5c121e58bafdfee51438cfe62a6cd72be735aef9753bfdfc7b27ecec
Sep 14 09:43:34.502052 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:34.502021    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66c47172_17e5_4985_84a8_987c216dd4ec.slice/crio-2827c36a1870a7c4c36cf18547d977afb320e38e91e9aa526c39a82e06573a5d WatchSource:0}: Error finding container 2827c36a1870a7c4c36cf18547d977afb320e38e91e9aa526c39a82e06573a5d: Status 404 returned error can't find the container with id 2827c36a1870a7c4c36cf18547d977afb320e38e91e9aa526c39a82e06573a5d
Sep 14 09:43:34.506509 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:34.506484    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddba3fdc7_1cc0_451e_9359_f597a2fb65a3.slice/crio-c94ad20b10773edc815783eb2764319fd25ce874ec7a87392029fc93cef2991f WatchSource:0}: Error finding container c94ad20b10773edc815783eb2764319fd25ce874ec7a87392029fc93cef2991f: Status 404 returned error can't find the container with id c94ad20b10773edc815783eb2764319fd25ce874ec7a87392029fc93cef2991f
Sep 14 09:43:34.508233 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:34.508200    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c80666a_ce28_47ef_b790_be0563101bb4.slice/crio-bdbedf2ddbac0b81bb28c6e664da74d4146f6e0c19448ab03c519b49b74f8613 WatchSource:0}: Error finding container bdbedf2ddbac0b81bb28c6e664da74d4146f6e0c19448ab03c519b49b74f8613: Status 404 returned error can't find the container with id bdbedf2ddbac0b81bb28c6e664da74d4146f6e0c19448ab03c519b49b74f8613
Sep 14 09:43:34.509492 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:34.509463    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda4b31da_fcbb_4132_bc49_1ecd91bb65bb.slice/crio-2d0cb7ed57a89880173ae91579ac515e955c7758ce4638d547dd25b5e94a5194 WatchSource:0}: Error finding container 2d0cb7ed57a89880173ae91579ac515e955c7758ce4638d547dd25b5e94a5194: Status 404 returned error can't find the container with id 2d0cb7ed57a89880173ae91579ac515e955c7758ce4638d547dd25b5e94a5194
Sep 14 09:43:34.510460 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:34.510399    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda9fcef7_654c_4140_bb22_abd2ece6e0c1.slice/crio-89de3864bac4c79c8168b07987d1754f5daa600b3b986d328e75f5e7fc194e71 WatchSource:0}: Error finding container 89de3864bac4c79c8168b07987d1754f5daa600b3b986d328e75f5e7fc194e71: Status 404 returned error can't find the container with id 89de3864bac4c79c8168b07987d1754f5daa600b3b986d328e75f5e7fc194e71
Sep 14 09:43:34.511604 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:43:34.511558    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c7467c2_fbc6_4195_b321_acc1ef713a13.slice/crio-c4ccdab1fd67110c7d4582cfeb56c2e3aa564f66d2b956aafd43fdd2e8af8e09 WatchSource:0}: Error finding container c4ccdab1fd67110c7d4582cfeb56c2e3aa564f66d2b956aafd43fdd2e8af8e09: Status 404 returned error can't find the container with id c4ccdab1fd67110c7d4582cfeb56c2e3aa564f66d2b956aafd43fdd2e8af8e09
Sep 14 09:43:35.057392 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:35.057347    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d956c018-b9cb-4c19-8f08-16652611d8c3-metrics-certs\") pod \"network-metrics-daemon-6l89k\" (UID: \"d956c018-b9cb-4c19-8f08-16652611d8c3\") " pod="openshift-multus/network-metrics-daemon-6l89k"
Sep 14 09:43:35.057605 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:35.057411    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97284aa2-9a80-495f-8b99-2ff7cec913da-original-pull-secret\") pod \"global-pull-secret-syncer-krg4v\" (UID: \"97284aa2-9a80-495f-8b99-2ff7cec913da\") " pod="kube-system/global-pull-secret-syncer-krg4v"
Sep 14 09:43:35.057605 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:35.057538    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 14 09:43:35.057605 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:35.057604    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97284aa2-9a80-495f-8b99-2ff7cec913da-original-pull-secret podName:97284aa2-9a80-495f-8b99-2ff7cec913da nodeName:}" failed. No retries permitted until 2026-09-14 09:43:37.057585864 +0000 UTC m=+6.097260477 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/97284aa2-9a80-495f-8b99-2ff7cec913da-original-pull-secret") pod "global-pull-secret-syncer-krg4v" (UID: "97284aa2-9a80-495f-8b99-2ff7cec913da") : object "kube-system"/"original-pull-secret" not registered
Sep 14 09:43:35.058076 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:35.058057    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 09:43:35.058160 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:35.058116    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d956c018-b9cb-4c19-8f08-16652611d8c3-metrics-certs podName:d956c018-b9cb-4c19-8f08-16652611d8c3 nodeName:}" failed. No retries permitted until 2026-09-14 09:43:37.058098137 +0000 UTC m=+6.097772750 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d956c018-b9cb-4c19-8f08-16652611d8c3-metrics-certs") pod "network-metrics-daemon-6l89k" (UID: "d956c018-b9cb-4c19-8f08-16652611d8c3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 09:43:35.158526 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:35.158483    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkm5l\" (UniqueName: \"kubernetes.io/projected/c41422ae-06c2-492c-9dbf-82e10222589a-kube-api-access-xkm5l\") pod \"network-check-target-8zcts\" (UID: \"c41422ae-06c2-492c-9dbf-82e10222589a\") " pod="openshift-network-diagnostics/network-check-target-8zcts"
Sep 14 09:43:35.158745 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:35.158725    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 14 09:43:35.158815 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:35.158756    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 14 09:43:35.158815 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:35.158770    2563 projected.go:196] Error preparing data for projected volume kube-api-access-xkm5l for pod openshift-network-diagnostics/network-check-target-8zcts: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 09:43:35.158952 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:35.158840    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c41422ae-06c2-492c-9dbf-82e10222589a-kube-api-access-xkm5l podName:c41422ae-06c2-492c-9dbf-82e10222589a nodeName:}" failed. No retries permitted until 2026-09-14 09:43:37.15882267 +0000 UTC m=+6.198497275 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-xkm5l" (UniqueName: "kubernetes.io/projected/c41422ae-06c2-492c-9dbf-82e10222589a-kube-api-access-xkm5l") pod "network-check-target-8zcts" (UID: "c41422ae-06c2-492c-9dbf-82e10222589a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 09:43:35.400330 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:35.400176    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-13 09:38:33 +0000 UTC" deadline="2028-04-27 23:29:54.062122716 +0000 UTC"
Sep 14 09:43:35.400330 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:35.400238    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14197h46m18.661889013s"
Sep 14 09:43:35.478351 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:35.478213    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6l89k"
Sep 14 09:43:35.478513 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:35.478374    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6l89k" podUID="d956c018-b9cb-4c19-8f08-16652611d8c3"
Sep 14 09:43:35.478830 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:35.478643    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8zcts"
Sep 14 09:43:35.478830 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:35.478212    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krg4v"
Sep 14 09:43:35.478830 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:35.478777    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8zcts" podUID="c41422ae-06c2-492c-9dbf-82e10222589a"
Sep 14 09:43:35.479067 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:35.478836    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-krg4v" podUID="97284aa2-9a80-495f-8b99-2ff7cec913da"
Sep 14 09:43:35.503561 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:35.502689    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-4.ec2.internal" event={"ID":"812604c7944c05f98e0873536f91322d","Type":"ContainerStarted","Data":"3dc0093d398b1e8a2d564546c831b07e1bdc87f53e3af5d13e1f7ba723bbf96d"}
Sep 14 09:43:35.526952 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:35.526907    2563 generic.go:356] "Generic (PLEG): container finished" podID="53b749d177a0ad18e520172c7d6ddac0" containerID="a2d21af44345aa714088541109f1ca4df168436eb3a595fb330427c7e1e7b307" exitCode=0
Sep 14 09:43:35.527878 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:35.527847    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal" event={"ID":"53b749d177a0ad18e520172c7d6ddac0","Type":"ContainerDied","Data":"a2d21af44345aa714088541109f1ca4df168436eb3a595fb330427c7e1e7b307"}
Sep 14 09:43:35.543966 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:35.543920    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-f5jts" event={"ID":"da4b31da-fcbb-4132-bc49-1ecd91bb65bb","Type":"ContainerStarted","Data":"2d0cb7ed57a89880173ae91579ac515e955c7758ce4638d547dd25b5e94a5194"}
Sep 14 09:43:35.546056 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:35.545735    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-4.ec2.internal" podStartSLOduration=3.545718227 podStartE2EDuration="3.545718227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 09:43:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:43:35.520559365 +0000 UTC m=+4.560233988" watchObservedRunningTime="2026-09-14 09:43:35.545718227 +0000 UTC m=+4.585392897"
Sep 14 09:43:35.556648 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:35.556605    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tcvsm" event={"ID":"1c80666a-ce28-47ef-b790-be0563101bb4","Type":"ContainerStarted","Data":"bdbedf2ddbac0b81bb28c6e664da74d4146f6e0c19448ab03c519b49b74f8613"}
Sep 14 09:43:35.575776 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:35.575737    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hdclt" event={"ID":"dba3fdc7-1cc0-451e-9359-f597a2fb65a3","Type":"ContainerStarted","Data":"c94ad20b10773edc815783eb2764319fd25ce874ec7a87392029fc93cef2991f"}
Sep 14 09:43:35.588882 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:35.588761    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjwpq" event={"ID":"0a7d1077-b7fb-479d-990d-e35e6e7611bd","Type":"ContainerStarted","Data":"5a3716a5f77827ebf350349bf6522d0f2a463cdeeee685bb595bb53d6df2fcd0"}
Sep 14 09:43:35.601730 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:35.601591    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cgxf2" event={"ID":"2c7467c2-fbc6-4195-b321-acc1ef713a13","Type":"ContainerStarted","Data":"c4ccdab1fd67110c7d4582cfeb56c2e3aa564f66d2b956aafd43fdd2e8af8e09"}
Sep 14 09:43:35.613916 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:35.613772    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kb4pz" event={"ID":"da9fcef7-654c-4140-bb22-abd2ece6e0c1","Type":"ContainerStarted","Data":"89de3864bac4c79c8168b07987d1754f5daa600b3b986d328e75f5e7fc194e71"}
Sep 14 09:43:35.619629 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:35.619548    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cskmj" event={"ID":"66c47172-17e5-4985-84a8-987c216dd4ec","Type":"ContainerStarted","Data":"2827c36a1870a7c4c36cf18547d977afb320e38e91e9aa526c39a82e06573a5d"}
Sep 14 09:43:35.646398 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:35.646351    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mq9n4" event={"ID":"07d5e1df-7d96-401d-b606-245e4b1f3ae5","Type":"ContainerStarted","Data":"1faf766f5c121e58bafdfee51438cfe62a6cd72be735aef9753bfdfc7b27ecec"}
Sep 14 09:43:36.672161 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:36.672084    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal" event={"ID":"53b749d177a0ad18e520172c7d6ddac0","Type":"ContainerStarted","Data":"249a77e314eec7785151ba33df18d6ba12d96599b122ce07261982eff39e696c"}
Sep 14 09:43:37.074657 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:37.074305    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97284aa2-9a80-495f-8b99-2ff7cec913da-original-pull-secret\") pod \"global-pull-secret-syncer-krg4v\" (UID: \"97284aa2-9a80-495f-8b99-2ff7cec913da\") " pod="kube-system/global-pull-secret-syncer-krg4v"
Sep 14 09:43:37.074657 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:37.074430    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d956c018-b9cb-4c19-8f08-16652611d8c3-metrics-certs\") pod \"network-metrics-daemon-6l89k\" (UID: \"d956c018-b9cb-4c19-8f08-16652611d8c3\") " pod="openshift-multus/network-metrics-daemon-6l89k"
Sep 14 09:43:37.074657 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:37.074545    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 09:43:37.074657 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:37.074547    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 14 09:43:37.074657 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:37.074612    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d956c018-b9cb-4c19-8f08-16652611d8c3-metrics-certs podName:d956c018-b9cb-4c19-8f08-16652611d8c3 nodeName:}" failed. No retries permitted until 2026-09-14 09:43:41.074591945 +0000 UTC m=+10.114266550 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d956c018-b9cb-4c19-8f08-16652611d8c3-metrics-certs") pod "network-metrics-daemon-6l89k" (UID: "d956c018-b9cb-4c19-8f08-16652611d8c3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 09:43:37.074657 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:37.074630    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97284aa2-9a80-495f-8b99-2ff7cec913da-original-pull-secret podName:97284aa2-9a80-495f-8b99-2ff7cec913da nodeName:}" failed. No retries permitted until 2026-09-14 09:43:41.074621399 +0000 UTC m=+10.114296003 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/97284aa2-9a80-495f-8b99-2ff7cec913da-original-pull-secret") pod "global-pull-secret-syncer-krg4v" (UID: "97284aa2-9a80-495f-8b99-2ff7cec913da") : object "kube-system"/"original-pull-secret" not registered
Sep 14 09:43:37.175771 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:37.175731    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkm5l\" (UniqueName: \"kubernetes.io/projected/c41422ae-06c2-492c-9dbf-82e10222589a-kube-api-access-xkm5l\") pod \"network-check-target-8zcts\" (UID: \"c41422ae-06c2-492c-9dbf-82e10222589a\") " pod="openshift-network-diagnostics/network-check-target-8zcts"
Sep 14 09:43:37.176021 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:37.175947    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 14 09:43:37.176021 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:37.175975    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 14 09:43:37.176021 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:37.175990    2563 projected.go:196] Error preparing data for projected volume kube-api-access-xkm5l for pod openshift-network-diagnostics/network-check-target-8zcts: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 09:43:37.176238 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:37.176078    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c41422ae-06c2-492c-9dbf-82e10222589a-kube-api-access-xkm5l podName:c41422ae-06c2-492c-9dbf-82e10222589a nodeName:}" failed. No retries permitted until 2026-09-14 09:43:41.176059716 +0000 UTC m=+10.215734333 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-xkm5l" (UniqueName: "kubernetes.io/projected/c41422ae-06c2-492c-9dbf-82e10222589a-kube-api-access-xkm5l") pod "network-check-target-8zcts" (UID: "c41422ae-06c2-492c-9dbf-82e10222589a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 09:43:37.476544 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:37.476456    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6l89k"
Sep 14 09:43:37.476544 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:37.476507    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krg4v"
Sep 14 09:43:37.476769 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:37.476456    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8zcts"
Sep 14 09:43:37.476769 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:37.476615    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6l89k" podUID="d956c018-b9cb-4c19-8f08-16652611d8c3"
Sep 14 09:43:37.476769 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:37.476691    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8zcts" podUID="c41422ae-06c2-492c-9dbf-82e10222589a"
Sep 14 09:43:37.476921 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:37.476778    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-krg4v" podUID="97284aa2-9a80-495f-8b99-2ff7cec913da"
Sep 14 09:43:39.475903 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:39.475866    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krg4v"
Sep 14 09:43:39.476405 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:39.476022    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-krg4v" podUID="97284aa2-9a80-495f-8b99-2ff7cec913da"
Sep 14 09:43:39.476405 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:39.476057    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6l89k"
Sep 14 09:43:39.476405 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:39.476176    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6l89k" podUID="d956c018-b9cb-4c19-8f08-16652611d8c3"
Sep 14 09:43:39.476405 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:39.475866    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8zcts"
Sep 14 09:43:39.476405 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:39.476267    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8zcts" podUID="c41422ae-06c2-492c-9dbf-82e10222589a"
Sep 14 09:43:41.109455 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:41.109408    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d956c018-b9cb-4c19-8f08-16652611d8c3-metrics-certs\") pod \"network-metrics-daemon-6l89k\" (UID: \"d956c018-b9cb-4c19-8f08-16652611d8c3\") " pod="openshift-multus/network-metrics-daemon-6l89k"
Sep 14 09:43:41.109908 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:41.109572    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 09:43:41.109908 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:41.109658    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d956c018-b9cb-4c19-8f08-16652611d8c3-metrics-certs podName:d956c018-b9cb-4c19-8f08-16652611d8c3 nodeName:}" failed. No retries permitted until 2026-09-14 09:43:49.109636563 +0000 UTC m=+18.149311167 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d956c018-b9cb-4c19-8f08-16652611d8c3-metrics-certs") pod "network-metrics-daemon-6l89k" (UID: "d956c018-b9cb-4c19-8f08-16652611d8c3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 09:43:41.109908 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:41.109685    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97284aa2-9a80-495f-8b99-2ff7cec913da-original-pull-secret\") pod \"global-pull-secret-syncer-krg4v\" (UID: \"97284aa2-9a80-495f-8b99-2ff7cec913da\") " pod="kube-system/global-pull-secret-syncer-krg4v"
Sep 14 09:43:41.109908 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:41.109858    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 14 09:43:41.110132 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:41.109932    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97284aa2-9a80-495f-8b99-2ff7cec913da-original-pull-secret podName:97284aa2-9a80-495f-8b99-2ff7cec913da nodeName:}" failed. No retries permitted until 2026-09-14 09:43:49.109901921 +0000 UTC m=+18.149576524 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/97284aa2-9a80-495f-8b99-2ff7cec913da-original-pull-secret") pod "global-pull-secret-syncer-krg4v" (UID: "97284aa2-9a80-495f-8b99-2ff7cec913da") : object "kube-system"/"original-pull-secret" not registered
Sep 14 09:43:41.210852 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:41.210770    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkm5l\" (UniqueName: \"kubernetes.io/projected/c41422ae-06c2-492c-9dbf-82e10222589a-kube-api-access-xkm5l\") pod \"network-check-target-8zcts\" (UID: \"c41422ae-06c2-492c-9dbf-82e10222589a\") " pod="openshift-network-diagnostics/network-check-target-8zcts"
Sep 14 09:43:41.211048 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:41.210971    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 14 09:43:41.211048 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:41.211013    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 14 09:43:41.211048 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:41.211029    2563 projected.go:196] Error preparing data for projected volume kube-api-access-xkm5l for pod openshift-network-diagnostics/network-check-target-8zcts: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 09:43:41.211232 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:41.211095    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c41422ae-06c2-492c-9dbf-82e10222589a-kube-api-access-xkm5l podName:c41422ae-06c2-492c-9dbf-82e10222589a nodeName:}" failed. No retries permitted until 2026-09-14 09:43:49.211074103 +0000 UTC m=+18.250748715 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-xkm5l" (UniqueName: "kubernetes.io/projected/c41422ae-06c2-492c-9dbf-82e10222589a-kube-api-access-xkm5l") pod "network-check-target-8zcts" (UID: "c41422ae-06c2-492c-9dbf-82e10222589a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 09:43:41.477029 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:41.476715    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6l89k"
Sep 14 09:43:41.477029 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:41.476860    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6l89k" podUID="d956c018-b9cb-4c19-8f08-16652611d8c3"
Sep 14 09:43:41.477255 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:41.477243    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8zcts"
Sep 14 09:43:41.477505 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:41.477341    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8zcts" podUID="c41422ae-06c2-492c-9dbf-82e10222589a"
Sep 14 09:43:41.477505 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:41.477396    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krg4v"
Sep 14 09:43:41.477505 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:41.477467    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-krg4v" podUID="97284aa2-9a80-495f-8b99-2ff7cec913da"
Sep 14 09:43:43.476082 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:43.475986    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6l89k"
Sep 14 09:43:43.476082 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:43.476041    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8zcts"
Sep 14 09:43:43.476572 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:43.476150    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krg4v"
Sep 14 09:43:43.476572 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:43.476157    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8zcts" podUID="c41422ae-06c2-492c-9dbf-82e10222589a"
Sep 14 09:43:43.476572 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:43.476253    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6l89k" podUID="d956c018-b9cb-4c19-8f08-16652611d8c3"
Sep 14 09:43:43.476572 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:43.476351    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-krg4v" podUID="97284aa2-9a80-495f-8b99-2ff7cec913da"
Sep 14 09:43:45.476070 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:45.476020    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krg4v"
Sep 14 09:43:45.476551 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:45.476020    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6l89k"
Sep 14 09:43:45.476551 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:45.476157    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-krg4v" podUID="97284aa2-9a80-495f-8b99-2ff7cec913da"
Sep 14 09:43:45.476551 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:45.476276    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6l89k" podUID="d956c018-b9cb-4c19-8f08-16652611d8c3"
Sep 14 09:43:45.476551 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:45.476044    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8zcts"
Sep 14 09:43:45.476551 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:45.476403    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8zcts" podUID="c41422ae-06c2-492c-9dbf-82e10222589a"
Sep 14 09:43:47.475967 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:47.475928    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krg4v"
Sep 14 09:43:47.476453 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:47.475928    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8zcts"
Sep 14 09:43:47.476453 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:47.476090    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-krg4v" podUID="97284aa2-9a80-495f-8b99-2ff7cec913da"
Sep 14 09:43:47.476453 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:47.475928    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6l89k"
Sep 14 09:43:47.476453 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:47.476189    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8zcts" podUID="c41422ae-06c2-492c-9dbf-82e10222589a"
Sep 14 09:43:47.476453 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:47.476242    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6l89k" podUID="d956c018-b9cb-4c19-8f08-16652611d8c3"
Sep 14 09:43:49.170011 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:49.169949    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97284aa2-9a80-495f-8b99-2ff7cec913da-original-pull-secret\") pod \"global-pull-secret-syncer-krg4v\" (UID: \"97284aa2-9a80-495f-8b99-2ff7cec913da\") " pod="kube-system/global-pull-secret-syncer-krg4v"
Sep 14 09:43:49.170570 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:49.170072    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d956c018-b9cb-4c19-8f08-16652611d8c3-metrics-certs\") pod \"network-metrics-daemon-6l89k\" (UID: \"d956c018-b9cb-4c19-8f08-16652611d8c3\") " pod="openshift-multus/network-metrics-daemon-6l89k"
Sep 14 09:43:49.170570 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:49.170151    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 14 09:43:49.170570 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:49.170185    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 09:43:49.170570 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:49.170237    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97284aa2-9a80-495f-8b99-2ff7cec913da-original-pull-secret podName:97284aa2-9a80-495f-8b99-2ff7cec913da nodeName:}" failed. No retries permitted until 2026-09-14 09:44:05.170214397 +0000 UTC m=+34.209889003 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/97284aa2-9a80-495f-8b99-2ff7cec913da-original-pull-secret") pod "global-pull-secret-syncer-krg4v" (UID: "97284aa2-9a80-495f-8b99-2ff7cec913da") : object "kube-system"/"original-pull-secret" not registered
Sep 14 09:43:49.170570 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:49.170260    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d956c018-b9cb-4c19-8f08-16652611d8c3-metrics-certs podName:d956c018-b9cb-4c19-8f08-16652611d8c3 nodeName:}" failed. No retries permitted until 2026-09-14 09:44:05.170248193 +0000 UTC m=+34.209922793 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d956c018-b9cb-4c19-8f08-16652611d8c3-metrics-certs") pod "network-metrics-daemon-6l89k" (UID: "d956c018-b9cb-4c19-8f08-16652611d8c3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 09:43:49.270705 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:49.270658    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkm5l\" (UniqueName: \"kubernetes.io/projected/c41422ae-06c2-492c-9dbf-82e10222589a-kube-api-access-xkm5l\") pod \"network-check-target-8zcts\" (UID: \"c41422ae-06c2-492c-9dbf-82e10222589a\") " pod="openshift-network-diagnostics/network-check-target-8zcts"
Sep 14 09:43:49.270912 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:49.270852    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 14 09:43:49.270912 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:49.270884    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 14 09:43:49.270912 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:49.270900    2563 projected.go:196] Error preparing data for projected volume kube-api-access-xkm5l for pod openshift-network-diagnostics/network-check-target-8zcts: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 09:43:49.271094 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:49.270971    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c41422ae-06c2-492c-9dbf-82e10222589a-kube-api-access-xkm5l podName:c41422ae-06c2-492c-9dbf-82e10222589a nodeName:}" failed. No retries permitted until 2026-09-14 09:44:05.270949595 +0000 UTC m=+34.310624197 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-xkm5l" (UniqueName: "kubernetes.io/projected/c41422ae-06c2-492c-9dbf-82e10222589a-kube-api-access-xkm5l") pod "network-check-target-8zcts" (UID: "c41422ae-06c2-492c-9dbf-82e10222589a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 09:43:49.476522 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:49.476419    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6l89k"
Sep 14 09:43:49.476522 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:49.476419    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krg4v"
Sep 14 09:43:49.476749 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:49.476424    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8zcts"
Sep 14 09:43:49.476749 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:49.476570    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6l89k" podUID="d956c018-b9cb-4c19-8f08-16652611d8c3"
Sep 14 09:43:49.476749 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:49.476705    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8zcts" podUID="c41422ae-06c2-492c-9dbf-82e10222589a"
Sep 14 09:43:49.476904 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:49.476787    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-krg4v" podUID="97284aa2-9a80-495f-8b99-2ff7cec913da"
Sep 14 09:43:51.476827 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:51.476786    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6l89k"
Sep 14 09:43:51.477344 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:51.476899    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8zcts"
Sep 14 09:43:51.477344 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:51.476953    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krg4v"
Sep 14 09:43:51.477344 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:51.476948    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6l89k" podUID="d956c018-b9cb-4c19-8f08-16652611d8c3"
Sep 14 09:43:51.477344 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:51.477055    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-krg4v" podUID="97284aa2-9a80-495f-8b99-2ff7cec913da"
Sep 14 09:43:51.477344 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:51.477142    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8zcts" podUID="c41422ae-06c2-492c-9dbf-82e10222589a"
Sep 14 09:43:52.708635 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:52.708258    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjwpq" event={"ID":"0a7d1077-b7fb-479d-990d-e35e6e7611bd","Type":"ContainerStarted","Data":"bac69db32eb4f5a086c1f89bbc4b10794cd7bd46a0d3343916442ab946fdc14b"}
Sep 14 09:43:52.709790 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:52.709766    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kb4pz" event={"ID":"da9fcef7-654c-4140-bb22-abd2ece6e0c1","Type":"ContainerStarted","Data":"8b80d961898137b3cf62f8b3c4629fd5a2a02497014a83729781c767498dddbd"}
Sep 14 09:43:52.711257 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:52.711227    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cskmj" event={"ID":"66c47172-17e5-4985-84a8-987c216dd4ec","Type":"ContainerStarted","Data":"0e38f801ed7761cc5c865798a78a9d6ee55b3bc567ac8a802f5c8d09f2c41956"}
Sep 14 09:43:52.712635 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:52.712614    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mq9n4" event={"ID":"07d5e1df-7d96-401d-b606-245e4b1f3ae5","Type":"ContainerStarted","Data":"1161a686b3e6ea402852335f018516508b6e62a560ec87094f7094f073998a74"}
Sep 14 09:43:52.714225 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:52.714194    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-f5jts" event={"ID":"da4b31da-fcbb-4132-bc49-1ecd91bb65bb","Type":"ContainerStarted","Data":"792a586e01409caef15e8db69787296d9f4a11f35797824843f57a78f8bb3c5f"}
Sep 14 09:43:52.715823 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:52.715798    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tcvsm" event={"ID":"1c80666a-ce28-47ef-b790-be0563101bb4","Type":"ContainerStarted","Data":"ffc03793169bb1d89b345bbb4675c6d436e0bda5d19b6842bd0a23d22083fbcc"}
Sep 14 09:43:52.718743 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:52.718713    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hdclt" event={"ID":"dba3fdc7-1cc0-451e-9359-f597a2fb65a3","Type":"ContainerStarted","Data":"94e7ef35bfe599efa0c8c5fadb452073c6675fe187453ba3762fc38d1187e77e"}
Sep 14 09:43:52.718878 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:52.718796    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hdclt" event={"ID":"dba3fdc7-1cc0-451e-9359-f597a2fb65a3","Type":"ContainerStarted","Data":"8ea715c31e324b6f9f0d925df26c9cb7c1826200d387702255bdcc7a0b54cdea"}
Sep 14 09:43:52.718878 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:52.718842    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hdclt" event={"ID":"dba3fdc7-1cc0-451e-9359-f597a2fb65a3","Type":"ContainerStarted","Data":"9440124e64cf3373e07c18688b285f402d37ea0bfe6420abf4b15e27ad1f9dd7"}
Sep 14 09:43:52.719020 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:52.718888    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hdclt" event={"ID":"dba3fdc7-1cc0-451e-9359-f597a2fb65a3","Type":"ContainerStarted","Data":"96e0f2d856c62421b1c576d475153e2a6b2d4f7a29650de146241fb879cea47e"}
Sep 14 09:43:52.719020 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:52.718940    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hdclt" event={"ID":"dba3fdc7-1cc0-451e-9359-f597a2fb65a3","Type":"ContainerStarted","Data":"3e98e6a295cb284a88f33754aaa22e29a9e49e8263467bf33c2f6960e32602f4"}
Sep 14 09:43:52.720335 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:52.720307    2563 generic.go:356] "Generic (PLEG): container finished" podID="20d9618f-664c-4cae-bcc2-4b00ba45acc6" containerID="8306ac4ff292e2d291c9a08f7b3afd72ccb6fcbc4189c0ea1f12527811f3300c" exitCode=0
Sep 14 09:43:52.720413 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:52.720344    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7tm8l" event={"ID":"20d9618f-664c-4cae-bcc2-4b00ba45acc6","Type":"ContainerDied","Data":"8306ac4ff292e2d291c9a08f7b3afd72ccb6fcbc4189c0ea1f12527811f3300c"}
Sep 14 09:43:52.732720 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:52.732670    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal" podStartSLOduration=20.732651902 podStartE2EDuration="20.732651902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 09:43:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:43:36.695814642 +0000 UTC m=+5.735489267" watchObservedRunningTime="2026-09-14 09:43:52.732651902 +0000 UTC m=+21.772326534"
Sep 14 09:43:52.751711 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:52.751648    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-kb4pz" podStartSLOduration=4.203869634 podStartE2EDuration="21.751627264s" totalImagesPullingTime="17.54775763s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 09:43:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:43:52.732737136 +0000 UTC m=+21.772411760" watchObservedRunningTime="2026-09-14 09:43:52.751627264 +0000 UTC m=+21.791301982"
Sep 14 09:43:52.752203 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:52.752174    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mq9n4" podStartSLOduration=4.193318285 podStartE2EDuration="21.752164832s" totalImagesPullingTime="17.558846547s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 09:43:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:43:52.751265074 +0000 UTC m=+21.790939696" watchObservedRunningTime="2026-09-14 09:43:52.752164832 +0000 UTC m=+21.791839454"
Sep 14 09:43:52.826219 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:52.825976    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-tcvsm" podStartSLOduration=4.218415068 podStartE2EDuration="21.825953736s" totalImagesPullingTime="17.607538668s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 09:43:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:43:52.789189607 +0000 UTC m=+21.828864231" watchObservedRunningTime="2026-09-14 09:43:52.825953736 +0000 UTC m=+21.865628360"
Sep 14 09:43:52.859736 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:52.859682    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-cskmj" podStartSLOduration=11.489592655 podStartE2EDuration="21.859659899s" totalImagesPullingTime="10.370067244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:43:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:43:52.826479148 +0000 UTC m=+21.866153761" watchObservedRunningTime="2026-09-14 09:43:52.859659899 +0000 UTC m=+21.899334521"
Sep 14 09:43:52.885184 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:52.884889    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-f5jts" podStartSLOduration=8.156622519 podStartE2EDuration="21.884867617s" totalImagesPullingTime="13.728245098s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 09:43:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:43:52.88436081 +0000 UTC m=+21.924035434" watchObservedRunningTime="2026-09-14 09:43:52.884867617 +0000 UTC m=+21.924542241"
Sep 14 09:43:52.927430 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:52.927390    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:43:53.055228 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:53.055188    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:43:53.154564 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:53.154528    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:43:53.476844 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:53.476738    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6l89k"
Sep 14 09:43:53.476844 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:53.476782    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krg4v"
Sep 14 09:43:53.477127 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:53.476890    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6l89k" podUID="d956c018-b9cb-4c19-8f08-16652611d8c3"
Sep 14 09:43:53.477431 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:53.476738    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8zcts"
Sep 14 09:43:53.477431 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:53.477307    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-krg4v" podUID="97284aa2-9a80-495f-8b99-2ff7cec913da"
Sep 14 09:43:53.477431 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:53.477391    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8zcts" podUID="c41422ae-06c2-492c-9dbf-82e10222589a"
Sep 14 09:43:53.549386 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:53.549308    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-cskmj"
Sep 14 09:43:53.549834 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:53.549813    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-cskmj"
Sep 14 09:43:53.563968 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:53.563932    2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 14 09:43:53.726080 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:53.725969    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hdclt" event={"ID":"dba3fdc7-1cc0-451e-9359-f597a2fb65a3","Type":"ContainerStarted","Data":"850a54b0b0b954129ccd8e01f7ad6cc6bdcf6995ef5b4ab9802bfdea030ff71d"}
Sep 14 09:43:53.727929 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:53.727898    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjwpq" event={"ID":"0a7d1077-b7fb-479d-990d-e35e6e7611bd","Type":"ContainerStarted","Data":"86e6bbe5c14155dc4bdcbd3c509150cbf023deda0b008ca9d32692b16346303d"}
Sep 14 09:43:53.729393 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:53.729339    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cgxf2" event={"ID":"2c7467c2-fbc6-4195-b321-acc1ef713a13","Type":"ContainerStarted","Data":"3a2da59fc8cb62c5e2978d37ab0214703b3888340f2dbcf3af49a86e0ae13268"}
Sep 14 09:43:53.755803 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:53.755743    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-cgxf2" podStartSLOduration=5.213360379 podStartE2EDuration="22.755726639s" totalImagesPullingTime="17.54236626s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 09:43:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:43:53.755604949 +0000 UTC m=+22.795279600" watchObservedRunningTime="2026-09-14 09:43:53.755726639 +0000 UTC m=+22.795401279"
Sep 14 09:43:53.943034 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:53.942986    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:43:54.031052 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:54.031014    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:43:54.421465 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:54.421293    2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-14T09:43:53.563964492Z","UUID":"7ef7971b-9508-47cb-8c71-8e75d402b69c","Handler":null,"Name":"","Endpoint":""}
Sep 14 09:43:54.422936 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:54.422912    2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 14 09:43:54.423096 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:54.422948    2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 14 09:43:54.734381 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:54.734356    2563 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 14 09:43:54.734381 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:54.734329    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjwpq" event={"ID":"0a7d1077-b7fb-479d-990d-e35e6e7611bd","Type":"ContainerStarted","Data":"ace51dad5435d8d9c8ce4143301ba8224972082479cb6eef3404b3549c06daca"}
Sep 14 09:43:54.755073 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:54.755018    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjwpq" podStartSLOduration=4.152203551 podStartE2EDuration="23.754977961s" totalImagesPullingTime="19.60277441s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 09:43:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:43:54.754554855 +0000 UTC m=+23.794229499" watchObservedRunningTime="2026-09-14 09:43:54.754977961 +0000 UTC m=+23.794652584"
Sep 14 09:43:55.476111 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:55.476075    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krg4v"
Sep 14 09:43:55.476329 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:55.476126    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6l89k"
Sep 14 09:43:55.476329 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:55.476216    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-krg4v" podUID="97284aa2-9a80-495f-8b99-2ff7cec913da"
Sep 14 09:43:55.476329 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:55.476226    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8zcts"
Sep 14 09:43:55.476494 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:55.476334    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6l89k" podUID="d956c018-b9cb-4c19-8f08-16652611d8c3"
Sep 14 09:43:55.476494 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:55.476451    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8zcts" podUID="c41422ae-06c2-492c-9dbf-82e10222589a"
Sep 14 09:43:55.744865 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:55.744816    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hdclt" event={"ID":"dba3fdc7-1cc0-451e-9359-f597a2fb65a3","Type":"ContainerStarted","Data":"533e4bc61597493b9ef57cdd8fad085db16faa9fd09039c4406c5fee74323972"}
Sep 14 09:43:57.476314 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:57.476277    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8zcts"
Sep 14 09:43:57.477186 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:57.476277    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6l89k"
Sep 14 09:43:57.477186 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:57.476393    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8zcts" podUID="c41422ae-06c2-492c-9dbf-82e10222589a"
Sep 14 09:43:57.477186 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:57.476477    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6l89k" podUID="d956c018-b9cb-4c19-8f08-16652611d8c3"
Sep 14 09:43:57.477186 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:57.476280    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krg4v"
Sep 14 09:43:57.477186 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:57.476550    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-krg4v" podUID="97284aa2-9a80-495f-8b99-2ff7cec913da"
Sep 14 09:43:57.753479 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:57.753438    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hdclt" event={"ID":"dba3fdc7-1cc0-451e-9359-f597a2fb65a3","Type":"ContainerStarted","Data":"e3aa11926923fb1336b64ed1e3e2af2e25ab1bbe9d873ed3981918ab432859ad"}
Sep 14 09:43:57.753757 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:57.753722    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hdclt"
Sep 14 09:43:57.755700 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:57.755666    2563 generic.go:356] "Generic (PLEG): container finished" podID="20d9618f-664c-4cae-bcc2-4b00ba45acc6" containerID="731fdfdab31996eaba43616be208e67b884db5554ec71aa5ff02248017b2989c" exitCode=0
Sep 14 09:43:57.755847 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:57.755727    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7tm8l" event={"ID":"20d9618f-664c-4cae-bcc2-4b00ba45acc6","Type":"ContainerDied","Data":"731fdfdab31996eaba43616be208e67b884db5554ec71aa5ff02248017b2989c"}
Sep 14 09:43:57.770909 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:57.770881    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hdclt"
Sep 14 09:43:57.793605 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:57.793546    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-hdclt" podStartSLOduration=9.188039815 podStartE2EDuration="26.793529976s" totalImagesPullingTime="17.605490161s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 09:43:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:43:57.793085049 +0000 UTC m=+26.832759673" watchObservedRunningTime="2026-09-14 09:43:57.793529976 +0000 UTC m=+26.833204597"
Sep 14 09:43:58.760633 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:58.760365    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7tm8l" event={"ID":"20d9618f-664c-4cae-bcc2-4b00ba45acc6","Type":"ContainerStarted","Data":"6ef25773943376631fc533fcd85db03fd166cd7381d8e56f6c9332a27e316d54"}
Sep 14 09:43:58.761068 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:58.760938    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hdclt"
Sep 14 09:43:58.761068 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:58.761017    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hdclt"
Sep 14 09:43:58.777440 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:58.777403    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hdclt"
Sep 14 09:43:59.154235 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:59.154197    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-krg4v"]
Sep 14 09:43:59.154387 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:59.154314    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krg4v"
Sep 14 09:43:59.154450 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:59.154408    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-krg4v" podUID="97284aa2-9a80-495f-8b99-2ff7cec913da"
Sep 14 09:43:59.160802 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:59.160721    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8zcts"]
Sep 14 09:43:59.160957 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:59.160833    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8zcts"
Sep 14 09:43:59.160957 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:59.160917    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8zcts" podUID="c41422ae-06c2-492c-9dbf-82e10222589a"
Sep 14 09:43:59.166891 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:59.166853    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6l89k"]
Sep 14 09:43:59.167097 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:59.167023    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6l89k"
Sep 14 09:43:59.167176 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:43:59.167152    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6l89k" podUID="d956c018-b9cb-4c19-8f08-16652611d8c3"
Sep 14 09:43:59.764815 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:59.764778    2563 generic.go:356] "Generic (PLEG): container finished" podID="20d9618f-664c-4cae-bcc2-4b00ba45acc6" containerID="6ef25773943376631fc533fcd85db03fd166cd7381d8e56f6c9332a27e316d54" exitCode=0
Sep 14 09:43:59.765327 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:43:59.764861    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7tm8l" event={"ID":"20d9618f-664c-4cae-bcc2-4b00ba45acc6","Type":"ContainerDied","Data":"6ef25773943376631fc533fcd85db03fd166cd7381d8e56f6c9332a27e316d54"}
Sep 14 09:44:00.476140 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:00.476106    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6l89k"
Sep 14 09:44:00.476334 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:00.476165    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8zcts"
Sep 14 09:44:00.476334 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:00.476266    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krg4v"
Sep 14 09:44:00.476334 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:00.476282    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6l89k" podUID="d956c018-b9cb-4c19-8f08-16652611d8c3"
Sep 14 09:44:00.476500 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:00.476376    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-krg4v" podUID="97284aa2-9a80-495f-8b99-2ff7cec913da"
Sep 14 09:44:00.476500 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:00.476476    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8zcts" podUID="c41422ae-06c2-492c-9dbf-82e10222589a"
Sep 14 09:44:00.770368 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:00.770335    2563 generic.go:356] "Generic (PLEG): container finished" podID="20d9618f-664c-4cae-bcc2-4b00ba45acc6" containerID="c11bbca357c33a46db6893d3523fc588ea1217c23d44ce0bc856a527e4e5fcc9" exitCode=0
Sep 14 09:44:00.770817 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:00.770420    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7tm8l" event={"ID":"20d9618f-664c-4cae-bcc2-4b00ba45acc6","Type":"ContainerDied","Data":"c11bbca357c33a46db6893d3523fc588ea1217c23d44ce0bc856a527e4e5fcc9"}
Sep 14 09:44:02.476247 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:02.476209    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8zcts"
Sep 14 09:44:02.476719 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:02.476261    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krg4v"
Sep 14 09:44:02.476719 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:02.476318    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6l89k"
Sep 14 09:44:02.476719 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:02.476337    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8zcts" podUID="c41422ae-06c2-492c-9dbf-82e10222589a"
Sep 14 09:44:02.476719 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:02.476435    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6l89k" podUID="d956c018-b9cb-4c19-8f08-16652611d8c3"
Sep 14 09:44:02.476719 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:02.476526    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-krg4v" podUID="97284aa2-9a80-495f-8b99-2ff7cec913da"
Sep 14 09:44:03.185644 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:03.185604    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:03.303148 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:03.303104    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-cskmj"
Sep 14 09:44:03.303341 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:03.303265    2563 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 14 09:44:03.303881 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:03.303858    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-cskmj"
Sep 14 09:44:03.345294 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:03.345247    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:03.486844 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:03.486803    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:03.705301 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:03.705056    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:04.475860 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:04.475827    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krg4v"
Sep 14 09:44:04.475860 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:04.475870    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8zcts"
Sep 14 09:44:04.476171 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:04.475827    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6l89k"
Sep 14 09:44:04.476171 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:04.475985    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-krg4v" podUID="97284aa2-9a80-495f-8b99-2ff7cec913da"
Sep 14 09:44:04.476171 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:04.476115    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6l89k" podUID="d956c018-b9cb-4c19-8f08-16652611d8c3"
Sep 14 09:44:04.476315 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:04.476209    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8zcts" podUID="c41422ae-06c2-492c-9dbf-82e10222589a"
Sep 14 09:44:05.197106 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:05.197071    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d956c018-b9cb-4c19-8f08-16652611d8c3-metrics-certs\") pod \"network-metrics-daemon-6l89k\" (UID: \"d956c018-b9cb-4c19-8f08-16652611d8c3\") " pod="openshift-multus/network-metrics-daemon-6l89k"
Sep 14 09:44:05.197106 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:05.197112    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97284aa2-9a80-495f-8b99-2ff7cec913da-original-pull-secret\") pod \"global-pull-secret-syncer-krg4v\" (UID: \"97284aa2-9a80-495f-8b99-2ff7cec913da\") " pod="kube-system/global-pull-secret-syncer-krg4v"
Sep 14 09:44:05.197584 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:05.197228    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 14 09:44:05.197584 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:05.197229    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 09:44:05.197584 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:05.197282    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97284aa2-9a80-495f-8b99-2ff7cec913da-original-pull-secret podName:97284aa2-9a80-495f-8b99-2ff7cec913da nodeName:}" failed. No retries permitted until 2026-09-14 09:44:37.197267945 +0000 UTC m=+66.236942544 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/97284aa2-9a80-495f-8b99-2ff7cec913da-original-pull-secret") pod "global-pull-secret-syncer-krg4v" (UID: "97284aa2-9a80-495f-8b99-2ff7cec913da") : object "kube-system"/"original-pull-secret" not registered
Sep 14 09:44:05.197584 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:05.197296    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d956c018-b9cb-4c19-8f08-16652611d8c3-metrics-certs podName:d956c018-b9cb-4c19-8f08-16652611d8c3 nodeName:}" failed. No retries permitted until 2026-09-14 09:44:37.197289257 +0000 UTC m=+66.236963856 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d956c018-b9cb-4c19-8f08-16652611d8c3-metrics-certs") pod "network-metrics-daemon-6l89k" (UID: "d956c018-b9cb-4c19-8f08-16652611d8c3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 09:44:05.298175 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:05.298136    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkm5l\" (UniqueName: \"kubernetes.io/projected/c41422ae-06c2-492c-9dbf-82e10222589a-kube-api-access-xkm5l\") pod \"network-check-target-8zcts\" (UID: \"c41422ae-06c2-492c-9dbf-82e10222589a\") " pod="openshift-network-diagnostics/network-check-target-8zcts"
Sep 14 09:44:05.298344 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:05.298270    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 14 09:44:05.298344 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:05.298286    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 14 09:44:05.298344 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:05.298295    2563 projected.go:196] Error preparing data for projected volume kube-api-access-xkm5l for pod openshift-network-diagnostics/network-check-target-8zcts: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 09:44:05.298449 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:05.298345    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c41422ae-06c2-492c-9dbf-82e10222589a-kube-api-access-xkm5l podName:c41422ae-06c2-492c-9dbf-82e10222589a nodeName:}" failed. No retries permitted until 2026-09-14 09:44:37.298332322 +0000 UTC m=+66.338006922 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-xkm5l" (UniqueName: "kubernetes.io/projected/c41422ae-06c2-492c-9dbf-82e10222589a-kube-api-access-xkm5l") pod "network-check-target-8zcts" (UID: "c41422ae-06c2-492c-9dbf-82e10222589a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 09:44:05.672437 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:05.672400    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeReady"
Sep 14 09:44:05.672604 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:05.672573    2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 14 09:44:05.732073 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:05.732038    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-mklrw"]
Sep 14 09:44:05.742584 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:05.742542    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-f7d9v"]
Sep 14 09:44:05.742753 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:05.742724    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mklrw"
Sep 14 09:44:05.745148 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:05.745107    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 14 09:44:05.745305 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:05.745118    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hwxbr\""
Sep 14 09:44:05.745305 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:05.745167    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 14 09:44:05.753596 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:05.752992    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-f7d9v"
Sep 14 09:44:05.754549 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:05.754451    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mklrw"]
Sep 14 09:44:05.754549 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:05.754520    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-f7d9v"]
Sep 14 09:44:05.756270 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:05.756098    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 14 09:44:05.756270 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:05.756157    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 14 09:44:05.756561 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:05.756542    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-wswml\""
Sep 14 09:44:05.757130 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:05.757110    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 14 09:44:05.787647 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:05.787617    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7tm8l" event={"ID":"20d9618f-664c-4cae-bcc2-4b00ba45acc6","Type":"ContainerStarted","Data":"c07d98301fe09a7d925aa2f33e12e4505d28a83ef8ce1005abbee662fe562171"}
Sep 14 09:44:05.893930 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:05.893887    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:05.901860 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:05.901824    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ec1015e1-7070-4f02-8f4e-2fd5d395c4e9-config-volume\") pod \"dns-default-mklrw\" (UID: \"ec1015e1-7070-4f02-8f4e-2fd5d395c4e9\") " pod="openshift-dns/dns-default-mklrw"
Sep 14 09:44:05.902024 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:05.901866    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-482x9\" (UniqueName: \"kubernetes.io/projected/ec1015e1-7070-4f02-8f4e-2fd5d395c4e9-kube-api-access-482x9\") pod \"dns-default-mklrw\" (UID: \"ec1015e1-7070-4f02-8f4e-2fd5d395c4e9\") " pod="openshift-dns/dns-default-mklrw"
Sep 14 09:44:05.902024 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:05.901916    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw4rc\" (UniqueName: \"kubernetes.io/projected/4a0bee38-c70e-420f-82f8-ff4f0c42e0a6-kube-api-access-jw4rc\") pod \"ingress-canary-f7d9v\" (UID: \"4a0bee38-c70e-420f-82f8-ff4f0c42e0a6\") " pod="openshift-ingress-canary/ingress-canary-f7d9v"
Sep 14 09:44:05.902145 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:05.902117    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec1015e1-7070-4f02-8f4e-2fd5d395c4e9-metrics-tls\") pod \"dns-default-mklrw\" (UID: \"ec1015e1-7070-4f02-8f4e-2fd5d395c4e9\") " pod="openshift-dns/dns-default-mklrw"
Sep 14 09:44:05.902191 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:05.902169    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec1015e1-7070-4f02-8f4e-2fd5d395c4e9-tmp-dir\") pod \"dns-default-mklrw\" (UID: \"ec1015e1-7070-4f02-8f4e-2fd5d395c4e9\") " pod="openshift-dns/dns-default-mklrw"
Sep 14 09:44:05.902191 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:05.902186    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4a0bee38-c70e-420f-82f8-ff4f0c42e0a6-cert\") pod \"ingress-canary-f7d9v\" (UID: \"4a0bee38-c70e-420f-82f8-ff4f0c42e0a6\") " pod="openshift-ingress-canary/ingress-canary-f7d9v"
Sep 14 09:44:05.994519 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:05.994473    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:06.003519 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:06.003480    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ec1015e1-7070-4f02-8f4e-2fd5d395c4e9-config-volume\") pod \"dns-default-mklrw\" (UID: \"ec1015e1-7070-4f02-8f4e-2fd5d395c4e9\") " pod="openshift-dns/dns-default-mklrw"
Sep 14 09:44:06.003519 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:06.003522    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-482x9\" (UniqueName: \"kubernetes.io/projected/ec1015e1-7070-4f02-8f4e-2fd5d395c4e9-kube-api-access-482x9\") pod \"dns-default-mklrw\" (UID: \"ec1015e1-7070-4f02-8f4e-2fd5d395c4e9\") " pod="openshift-dns/dns-default-mklrw"
Sep 14 09:44:06.003684 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:06.003547    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw4rc\" (UniqueName: \"kubernetes.io/projected/4a0bee38-c70e-420f-82f8-ff4f0c42e0a6-kube-api-access-jw4rc\") pod \"ingress-canary-f7d9v\" (UID: \"4a0bee38-c70e-420f-82f8-ff4f0c42e0a6\") " pod="openshift-ingress-canary/ingress-canary-f7d9v"
Sep 14 09:44:06.003684 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:06.003599    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec1015e1-7070-4f02-8f4e-2fd5d395c4e9-metrics-tls\") pod \"dns-default-mklrw\" (UID: \"ec1015e1-7070-4f02-8f4e-2fd5d395c4e9\") " pod="openshift-dns/dns-default-mklrw"
Sep 14 09:44:06.003684 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:06.003617    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec1015e1-7070-4f02-8f4e-2fd5d395c4e9-tmp-dir\") pod \"dns-default-mklrw\" (UID: \"ec1015e1-7070-4f02-8f4e-2fd5d395c4e9\") " pod="openshift-dns/dns-default-mklrw"
Sep 14 09:44:06.003684 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:06.003636    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4a0bee38-c70e-420f-82f8-ff4f0c42e0a6-cert\") pod \"ingress-canary-f7d9v\" (UID: \"4a0bee38-c70e-420f-82f8-ff4f0c42e0a6\") " pod="openshift-ingress-canary/ingress-canary-f7d9v"
Sep 14 09:44:06.003886 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:06.003822    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 14 09:44:06.003950 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:06.003906    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec1015e1-7070-4f02-8f4e-2fd5d395c4e9-metrics-tls podName:ec1015e1-7070-4f02-8f4e-2fd5d395c4e9 nodeName:}" failed. No retries permitted until 2026-09-14 09:44:06.503883853 +0000 UTC m=+35.543558470 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ec1015e1-7070-4f02-8f4e-2fd5d395c4e9-metrics-tls") pod "dns-default-mklrw" (UID: "ec1015e1-7070-4f02-8f4e-2fd5d395c4e9") : secret "dns-default-metrics-tls" not found
Sep 14 09:44:06.004065 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:06.004037    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec1015e1-7070-4f02-8f4e-2fd5d395c4e9-tmp-dir\") pod \"dns-default-mklrw\" (UID: \"ec1015e1-7070-4f02-8f4e-2fd5d395c4e9\") " pod="openshift-dns/dns-default-mklrw"
Sep 14 09:44:06.004228 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:06.004154    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ec1015e1-7070-4f02-8f4e-2fd5d395c4e9-config-volume\") pod \"dns-default-mklrw\" (UID: \"ec1015e1-7070-4f02-8f4e-2fd5d395c4e9\") " pod="openshift-dns/dns-default-mklrw"
Sep 14 09:44:06.008220 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:06.008183    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4a0bee38-c70e-420f-82f8-ff4f0c42e0a6-cert\") pod \"ingress-canary-f7d9v\" (UID: \"4a0bee38-c70e-420f-82f8-ff4f0c42e0a6\") " pod="openshift-ingress-canary/ingress-canary-f7d9v"
Sep 14 09:44:06.024178 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:06.024141    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-482x9\" (UniqueName: \"kubernetes.io/projected/ec1015e1-7070-4f02-8f4e-2fd5d395c4e9-kube-api-access-482x9\") pod \"dns-default-mklrw\" (UID: \"ec1015e1-7070-4f02-8f4e-2fd5d395c4e9\") " pod="openshift-dns/dns-default-mklrw"
Sep 14 09:44:06.024951 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:06.024923    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw4rc\" (UniqueName: \"kubernetes.io/projected/4a0bee38-c70e-420f-82f8-ff4f0c42e0a6-kube-api-access-jw4rc\") pod \"ingress-canary-f7d9v\" (UID: \"4a0bee38-c70e-420f-82f8-ff4f0c42e0a6\") " pod="openshift-ingress-canary/ingress-canary-f7d9v"
Sep 14 09:44:06.065738 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:06.065632    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-f7d9v"
Sep 14 09:44:06.103535 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:06.103503    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:06.398018 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:06.397904    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-f7d9v"]
Sep 14 09:44:06.402189 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:44:06.402153    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a0bee38_c70e_420f_82f8_ff4f0c42e0a6.slice/crio-e48fc67e870e9ad2548ed64f1d7895f3c078888e4a9ba8cd529694d7d4177b4a WatchSource:0}: Error finding container e48fc67e870e9ad2548ed64f1d7895f3c078888e4a9ba8cd529694d7d4177b4a: Status 404 returned error can't find the container with id e48fc67e870e9ad2548ed64f1d7895f3c078888e4a9ba8cd529694d7d4177b4a
Sep 14 09:44:06.476081 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:06.476041    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8zcts"
Sep 14 09:44:06.476274 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:06.476038    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6l89k"
Sep 14 09:44:06.476274 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:06.476054    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krg4v"
Sep 14 09:44:06.479437 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:06.479412    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-clrv6\""
Sep 14 09:44:06.479587 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:06.479447    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 14 09:44:06.479718 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:06.479701    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 14 09:44:06.479777 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:06.479701    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 14 09:44:06.479825 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:06.479776    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 14 09:44:06.479927 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:06.479912    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9ctbn\""
Sep 14 09:44:06.507015 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:06.506963    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec1015e1-7070-4f02-8f4e-2fd5d395c4e9-metrics-tls\") pod \"dns-default-mklrw\" (UID: \"ec1015e1-7070-4f02-8f4e-2fd5d395c4e9\") " pod="openshift-dns/dns-default-mklrw"
Sep 14 09:44:06.507170 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:06.507123    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 14 09:44:06.507261 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:06.507211    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec1015e1-7070-4f02-8f4e-2fd5d395c4e9-metrics-tls podName:ec1015e1-7070-4f02-8f4e-2fd5d395c4e9 nodeName:}" failed. No retries permitted until 2026-09-14 09:44:07.507188859 +0000 UTC m=+36.546863473 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ec1015e1-7070-4f02-8f4e-2fd5d395c4e9-metrics-tls") pod "dns-default-mklrw" (UID: "ec1015e1-7070-4f02-8f4e-2fd5d395c4e9") : secret "dns-default-metrics-tls" not found
Sep 14 09:44:06.791318 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:06.791277    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-f7d9v" event={"ID":"4a0bee38-c70e-420f-82f8-ff4f0c42e0a6","Type":"ContainerStarted","Data":"e48fc67e870e9ad2548ed64f1d7895f3c078888e4a9ba8cd529694d7d4177b4a"}
Sep 14 09:44:06.794307 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:06.794272    2563 generic.go:356] "Generic (PLEG): container finished" podID="20d9618f-664c-4cae-bcc2-4b00ba45acc6" containerID="c07d98301fe09a7d925aa2f33e12e4505d28a83ef8ce1005abbee662fe562171" exitCode=0
Sep 14 09:44:06.794484 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:06.794330    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7tm8l" event={"ID":"20d9618f-664c-4cae-bcc2-4b00ba45acc6","Type":"ContainerDied","Data":"c07d98301fe09a7d925aa2f33e12e4505d28a83ef8ce1005abbee662fe562171"}
Sep 14 09:44:07.513619 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:07.513384    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec1015e1-7070-4f02-8f4e-2fd5d395c4e9-metrics-tls\") pod \"dns-default-mklrw\" (UID: \"ec1015e1-7070-4f02-8f4e-2fd5d395c4e9\") " pod="openshift-dns/dns-default-mklrw"
Sep 14 09:44:07.514114 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:07.513535    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 14 09:44:07.514114 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:07.513704    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec1015e1-7070-4f02-8f4e-2fd5d395c4e9-metrics-tls podName:ec1015e1-7070-4f02-8f4e-2fd5d395c4e9 nodeName:}" failed. No retries permitted until 2026-09-14 09:44:09.513683737 +0000 UTC m=+38.553358354 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ec1015e1-7070-4f02-8f4e-2fd5d395c4e9-metrics-tls") pod "dns-default-mklrw" (UID: "ec1015e1-7070-4f02-8f4e-2fd5d395c4e9") : secret "dns-default-metrics-tls" not found
Sep 14 09:44:07.799859 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:07.799774    2563 generic.go:356] "Generic (PLEG): container finished" podID="20d9618f-664c-4cae-bcc2-4b00ba45acc6" containerID="5aaf566a494dd117cca10185bbb8085bcef6c96ca0dfeca5277f095c4fc79279" exitCode=0
Sep 14 09:44:07.800015 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:07.799850    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7tm8l" event={"ID":"20d9618f-664c-4cae-bcc2-4b00ba45acc6","Type":"ContainerDied","Data":"5aaf566a494dd117cca10185bbb8085bcef6c96ca0dfeca5277f095c4fc79279"}
Sep 14 09:44:08.805672 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:08.805630    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7tm8l" event={"ID":"20d9618f-664c-4cae-bcc2-4b00ba45acc6","Type":"ContainerStarted","Data":"8582fa5f4ce00c077c95560d7698ced8445f604ae4cfa80761e5f2ab799f2239"}
Sep 14 09:44:08.851233 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:08.851173    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-7tm8l" podStartSLOduration=9.350702522 podStartE2EDuration="37.851157504s" totalImagesPullingTime="28.500454982s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 09:43:31 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:44:08.849565693 +0000 UTC m=+37.889240315" watchObservedRunningTime="2026-09-14 09:44:08.851157504 +0000 UTC m=+37.890832125"
Sep 14 09:44:09.019438 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:09.019410    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:09.134878 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:09.134826    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:09.266932 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:09.266851    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:09.529334 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:09.529245    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec1015e1-7070-4f02-8f4e-2fd5d395c4e9-metrics-tls\") pod \"dns-default-mklrw\" (UID: \"ec1015e1-7070-4f02-8f4e-2fd5d395c4e9\") " pod="openshift-dns/dns-default-mklrw"
Sep 14 09:44:09.529496 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:09.529360    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 14 09:44:09.529496 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:09.529416    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec1015e1-7070-4f02-8f4e-2fd5d395c4e9-metrics-tls podName:ec1015e1-7070-4f02-8f4e-2fd5d395c4e9 nodeName:}" failed. No retries permitted until 2026-09-14 09:44:13.529402483 +0000 UTC m=+42.569077082 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ec1015e1-7070-4f02-8f4e-2fd5d395c4e9-metrics-tls") pod "dns-default-mklrw" (UID: "ec1015e1-7070-4f02-8f4e-2fd5d395c4e9") : secret "dns-default-metrics-tls" not found
Sep 14 09:44:09.809902 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:09.809799    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-f7d9v" event={"ID":"4a0bee38-c70e-420f-82f8-ff4f0c42e0a6","Type":"ContainerStarted","Data":"e66278c4ba30cfb53c9a8d1e380c82f4b4a5056859423dcf682fedc5235db170"}
Sep 14 09:44:09.844892 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:09.844841    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-f7d9v" podStartSLOduration=2.241351929 podStartE2EDuration="4.844824771s" totalImagesPullingTime="2.603472842s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:44:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:44:09.844171726 +0000 UTC m=+38.883846348" watchObservedRunningTime="2026-09-14 09:44:09.844824771 +0000 UTC m=+38.884499392"
Sep 14 09:44:10.881736 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:10.881698    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:10.974181 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:10.974146    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:13.560107 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:13.560066    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec1015e1-7070-4f02-8f4e-2fd5d395c4e9-metrics-tls\") pod \"dns-default-mklrw\" (UID: \"ec1015e1-7070-4f02-8f4e-2fd5d395c4e9\") " pod="openshift-dns/dns-default-mklrw"
Sep 14 09:44:13.560509 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:13.560219    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 14 09:44:13.560509 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:13.560286    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec1015e1-7070-4f02-8f4e-2fd5d395c4e9-metrics-tls podName:ec1015e1-7070-4f02-8f4e-2fd5d395c4e9 nodeName:}" failed. No retries permitted until 2026-09-14 09:44:21.560269553 +0000 UTC m=+50.599944151 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ec1015e1-7070-4f02-8f4e-2fd5d395c4e9-metrics-tls") pod "dns-default-mklrw" (UID: "ec1015e1-7070-4f02-8f4e-2fd5d395c4e9") : secret "dns-default-metrics-tls" not found
Sep 14 09:44:21.618301 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:21.618257    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec1015e1-7070-4f02-8f4e-2fd5d395c4e9-metrics-tls\") pod \"dns-default-mklrw\" (UID: \"ec1015e1-7070-4f02-8f4e-2fd5d395c4e9\") " pod="openshift-dns/dns-default-mklrw"
Sep 14 09:44:21.618870 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:21.618413    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 14 09:44:21.618870 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:21.618481    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec1015e1-7070-4f02-8f4e-2fd5d395c4e9-metrics-tls podName:ec1015e1-7070-4f02-8f4e-2fd5d395c4e9 nodeName:}" failed. No retries permitted until 2026-09-14 09:44:37.618463856 +0000 UTC m=+66.658138455 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ec1015e1-7070-4f02-8f4e-2fd5d395c4e9-metrics-tls") pod "dns-default-mklrw" (UID: "ec1015e1-7070-4f02-8f4e-2fd5d395c4e9") : secret "dns-default-metrics-tls" not found
Sep 14 09:44:30.781781 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:30.781750    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hdclt"
Sep 14 09:44:30.872481 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:30.872441    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-7tpzw"]
Sep 14 09:44:30.875183 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:30.875160    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-7tpzw"
Sep 14 09:44:30.879330 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:30.879296    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-nvtsx\""
Sep 14 09:44:30.879828 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:30.879808    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 14 09:44:30.882491 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:30.882033    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 14 09:44:30.882491 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:30.882329    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 14 09:44:30.882830 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:30.882814    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 14 09:44:30.917629 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:30.917597    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-7tpzw"]
Sep 14 09:44:30.987592 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:30.987548    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zbjj\" (UniqueName: \"kubernetes.io/projected/161e5bed-e6f8-4c08-8c69-857064cc8c4f-kube-api-access-8zbjj\") pod \"cluster-monitoring-operator-6cd954b897-7tpzw\" (UID: \"161e5bed-e6f8-4c08-8c69-857064cc8c4f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-7tpzw"
Sep 14 09:44:30.987592 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:30.987597    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/161e5bed-e6f8-4c08-8c69-857064cc8c4f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-7tpzw\" (UID: \"161e5bed-e6f8-4c08-8c69-857064cc8c4f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-7tpzw"
Sep 14 09:44:30.987803 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:30.987658    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/161e5bed-e6f8-4c08-8c69-857064cc8c4f-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-7tpzw\" (UID: \"161e5bed-e6f8-4c08-8c69-857064cc8c4f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-7tpzw"
Sep 14 09:44:31.088600 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:31.088504    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/161e5bed-e6f8-4c08-8c69-857064cc8c4f-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-7tpzw\" (UID: \"161e5bed-e6f8-4c08-8c69-857064cc8c4f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-7tpzw"
Sep 14 09:44:31.088600 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:31.088579    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zbjj\" (UniqueName: \"kubernetes.io/projected/161e5bed-e6f8-4c08-8c69-857064cc8c4f-kube-api-access-8zbjj\") pod \"cluster-monitoring-operator-6cd954b897-7tpzw\" (UID: \"161e5bed-e6f8-4c08-8c69-857064cc8c4f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-7tpzw"
Sep 14 09:44:31.088600 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:31.088600    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/161e5bed-e6f8-4c08-8c69-857064cc8c4f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-7tpzw\" (UID: \"161e5bed-e6f8-4c08-8c69-857064cc8c4f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-7tpzw"
Sep 14 09:44:31.088835 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:31.088693    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 14 09:44:31.088835 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:31.088748    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/161e5bed-e6f8-4c08-8c69-857064cc8c4f-cluster-monitoring-operator-tls podName:161e5bed-e6f8-4c08-8c69-857064cc8c4f nodeName:}" failed. No retries permitted until 2026-09-14 09:44:31.588733109 +0000 UTC m=+60.628407708 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/161e5bed-e6f8-4c08-8c69-857064cc8c4f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-7tpzw" (UID: "161e5bed-e6f8-4c08-8c69-857064cc8c4f") : secret "cluster-monitoring-operator-tls" not found
Sep 14 09:44:31.089275 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:31.089255    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/161e5bed-e6f8-4c08-8c69-857064cc8c4f-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-7tpzw\" (UID: \"161e5bed-e6f8-4c08-8c69-857064cc8c4f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-7tpzw"
Sep 14 09:44:31.188119 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:31.188087    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zbjj\" (UniqueName: \"kubernetes.io/projected/161e5bed-e6f8-4c08-8c69-857064cc8c4f-kube-api-access-8zbjj\") pod \"cluster-monitoring-operator-6cd954b897-7tpzw\" (UID: \"161e5bed-e6f8-4c08-8c69-857064cc8c4f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-7tpzw"
Sep 14 09:44:31.219923 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:31.219885    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:31.428849 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:31.428761    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-9689ffdd5-8254l"]
Sep 14 09:44:31.430707 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:31.430689    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-9689ffdd5-8254l"
Sep 14 09:44:31.434867 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:31.434840    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 14 09:44:31.434867 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:31.434862    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 14 09:44:31.435123 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:31.434908    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xchsg\""
Sep 14 09:44:31.444824 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:31.444792    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 14 09:44:31.448863 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:31.448809    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-pdhmm"]
Sep 14 09:44:31.449561 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:31.449545    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 14 09:44:31.450652 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:31.450635    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-pdhmm"
Sep 14 09:44:31.462501 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:31.462468    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 14 09:44:31.462501 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:31.462486    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 14 09:44:31.463316 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:31.463297    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 14 09:44:31.464591 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:31.464573    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-6kkkf\""
Sep 14 09:44:31.480806 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:31.480775    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-9689ffdd5-8254l"]
Sep 14 09:44:31.485475 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:31.485441    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-pdhmm"]
Sep 14 09:44:31.492984 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:31.492951    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8-registry-certificates\") pod \"image-registry-9689ffdd5-8254l\" (UID: \"cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8\") " pod="openshift-image-registry/image-registry-9689ffdd5-8254l"
Sep 14 09:44:31.493084 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:31.492988    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8-ca-trust-extracted\") pod \"image-registry-9689ffdd5-8254l\" (UID: \"cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8\") " pod="openshift-image-registry/image-registry-9689ffdd5-8254l"
Sep 14 09:44:31.493084 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:31.493033    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8-bound-sa-token\") pod \"image-registry-9689ffdd5-8254l\" (UID: \"cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8\") " pod="openshift-image-registry/image-registry-9689ffdd5-8254l"
Sep 14 09:44:31.493084 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:31.493068    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8-registry-tls\") pod \"image-registry-9689ffdd5-8254l\" (UID: \"cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8\") " pod="openshift-image-registry/image-registry-9689ffdd5-8254l"
Sep 14 09:44:31.493181 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:31.493091    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs782\" (UniqueName: \"kubernetes.io/projected/cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8-kube-api-access-cs782\") pod \"image-registry-9689ffdd5-8254l\" (UID: \"cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8\") " pod="openshift-image-registry/image-registry-9689ffdd5-8254l"
Sep 14 09:44:31.493181 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:31.493107    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4e19379f-15ef-494e-9760-8087d2a910ba-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-pdhmm\" (UID: \"4e19379f-15ef-494e-9760-8087d2a910ba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-pdhmm"
Sep 14 09:44:31.493181 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:31.493128    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fkrh\" (UniqueName: \"kubernetes.io/projected/4e19379f-15ef-494e-9760-8087d2a910ba-kube-api-access-6fkrh\") pod \"cluster-samples-operator-86499c89c4-pdhmm\" (UID: \"4e19379f-15ef-494e-9760-8087d2a910ba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-pdhmm"
Sep 14 09:44:31.493269 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:31.493202    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8-image-registry-private-configuration\") pod \"image-registry-9689ffdd5-8254l\" (UID: \"cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8\") " pod="openshift-image-registry/image-registry-9689ffdd5-8254l"
Sep 14 09:44:31.493269 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:31.493224    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8-trusted-ca\") pod \"image-registry-9689ffdd5-8254l\" (UID: \"cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8\") " pod="openshift-image-registry/image-registry-9689ffdd5-8254l"
Sep 14 09:44:31.493269 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:31.493253    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8-installation-pull-secrets\") pod \"image-registry-9689ffdd5-8254l\" (UID: \"cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8\") " pod="openshift-image-registry/image-registry-9689ffdd5-8254l"
Sep 14 09:44:31.516625 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:31.516569    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:31.593839 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:31.593796    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8-registry-certificates\") pod \"image-registry-9689ffdd5-8254l\" (UID: \"cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8\") " pod="openshift-image-registry/image-registry-9689ffdd5-8254l"
Sep 14 09:44:31.593839 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:31.593837    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8-ca-trust-extracted\") pod \"image-registry-9689ffdd5-8254l\" (UID: \"cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8\") " pod="openshift-image-registry/image-registry-9689ffdd5-8254l"
Sep 14 09:44:31.594148 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:31.593863    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8-bound-sa-token\") pod \"image-registry-9689ffdd5-8254l\" (UID: \"cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8\") " pod="openshift-image-registry/image-registry-9689ffdd5-8254l"
Sep 14 09:44:31.594148 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:31.594033    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8-registry-tls\") pod \"image-registry-9689ffdd5-8254l\" (UID: \"cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8\") " pod="openshift-image-registry/image-registry-9689ffdd5-8254l"
Sep 14 09:44:31.594148 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:31.594089    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs782\" (UniqueName: \"kubernetes.io/projected/cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8-kube-api-access-cs782\") pod \"image-registry-9689ffdd5-8254l\" (UID: \"cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8\") " pod="openshift-image-registry/image-registry-9689ffdd5-8254l"
Sep 14 09:44:31.594148 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:31.594117    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4e19379f-15ef-494e-9760-8087d2a910ba-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-pdhmm\" (UID: \"4e19379f-15ef-494e-9760-8087d2a910ba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-pdhmm"
Sep 14 09:44:31.594148 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:31.594120    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 14 09:44:31.594148 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:31.594138    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9689ffdd5-8254l: secret "image-registry-tls" not found
Sep 14 09:44:31.594148 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:31.594143    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fkrh\" (UniqueName: \"kubernetes.io/projected/4e19379f-15ef-494e-9760-8087d2a910ba-kube-api-access-6fkrh\") pod \"cluster-samples-operator-86499c89c4-pdhmm\" (UID: \"4e19379f-15ef-494e-9760-8087d2a910ba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-pdhmm"
Sep 14 09:44:31.594475 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:31.594178    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/161e5bed-e6f8-4c08-8c69-857064cc8c4f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-7tpzw\" (UID: \"161e5bed-e6f8-4c08-8c69-857064cc8c4f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-7tpzw"
Sep 14 09:44:31.594475 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:31.594211    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8-registry-tls podName:cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8 nodeName:}" failed. No retries permitted until 2026-09-14 09:44:32.094191224 +0000 UTC m=+61.133865841 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8-registry-tls") pod "image-registry-9689ffdd5-8254l" (UID: "cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8") : secret "image-registry-tls" not found
Sep 14 09:44:31.594475 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:31.594257    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 14 09:44:31.594475 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:31.594272    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8-image-registry-private-configuration\") pod \"image-registry-9689ffdd5-8254l\" (UID: \"cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8\") " pod="openshift-image-registry/image-registry-9689ffdd5-8254l"
Sep 14 09:44:31.594475 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:31.594299    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8-trusted-ca\") pod \"image-registry-9689ffdd5-8254l\" (UID: \"cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8\") " pod="openshift-image-registry/image-registry-9689ffdd5-8254l"
Sep 14 09:44:31.594475 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:31.594328    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e19379f-15ef-494e-9760-8087d2a910ba-samples-operator-tls podName:4e19379f-15ef-494e-9760-8087d2a910ba nodeName:}" failed. No retries permitted until 2026-09-14 09:44:32.094311423 +0000 UTC m=+61.133986026 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4e19379f-15ef-494e-9760-8087d2a910ba-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-pdhmm" (UID: "4e19379f-15ef-494e-9760-8087d2a910ba") : secret "samples-operator-tls" not found
Sep 14 09:44:31.594475 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:31.594262    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 14 09:44:31.594475 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:31.594382    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8-installation-pull-secrets\") pod \"image-registry-9689ffdd5-8254l\" (UID: \"cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8\") " pod="openshift-image-registry/image-registry-9689ffdd5-8254l"
Sep 14 09:44:31.594475 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:31.594398    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/161e5bed-e6f8-4c08-8c69-857064cc8c4f-cluster-monitoring-operator-tls podName:161e5bed-e6f8-4c08-8c69-857064cc8c4f nodeName:}" failed. No retries permitted until 2026-09-14 09:44:32.594383453 +0000 UTC m=+61.634058053 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/161e5bed-e6f8-4c08-8c69-857064cc8c4f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-7tpzw" (UID: "161e5bed-e6f8-4c08-8c69-857064cc8c4f") : secret "cluster-monitoring-operator-tls" not found
Sep 14 09:44:31.594475 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:31.594335    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8-ca-trust-extracted\") pod \"image-registry-9689ffdd5-8254l\" (UID: \"cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8\") " pod="openshift-image-registry/image-registry-9689ffdd5-8254l"
Sep 14 09:44:31.594878 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:31.594582    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8-registry-certificates\") pod \"image-registry-9689ffdd5-8254l\" (UID: \"cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8\") " pod="openshift-image-registry/image-registry-9689ffdd5-8254l"
Sep 14 09:44:31.595032 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:31.594993    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8-trusted-ca\") pod \"image-registry-9689ffdd5-8254l\" (UID: \"cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8\") " pod="openshift-image-registry/image-registry-9689ffdd5-8254l"
Sep 14 09:44:31.596693 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:31.596668    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8-image-registry-private-configuration\") pod \"image-registry-9689ffdd5-8254l\" (UID: \"cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8\") " pod="openshift-image-registry/image-registry-9689ffdd5-8254l"
Sep 14 09:44:31.596797 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:31.596740    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8-installation-pull-secrets\") pod \"image-registry-9689ffdd5-8254l\" (UID: \"cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8\") " pod="openshift-image-registry/image-registry-9689ffdd5-8254l"
Sep 14 09:44:31.651862 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:31.651823    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fkrh\" (UniqueName: \"kubernetes.io/projected/4e19379f-15ef-494e-9760-8087d2a910ba-kube-api-access-6fkrh\") pod \"cluster-samples-operator-86499c89c4-pdhmm\" (UID: \"4e19379f-15ef-494e-9760-8087d2a910ba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-pdhmm"
Sep 14 09:44:31.652492 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:31.652467    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8-bound-sa-token\") pod \"image-registry-9689ffdd5-8254l\" (UID: \"cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8\") " pod="openshift-image-registry/image-registry-9689ffdd5-8254l"
Sep 14 09:44:31.660868 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:31.660833    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs782\" (UniqueName: \"kubernetes.io/projected/cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8-kube-api-access-cs782\") pod \"image-registry-9689ffdd5-8254l\" (UID: \"cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8\") " pod="openshift-image-registry/image-registry-9689ffdd5-8254l"
Sep 14 09:44:31.861888 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:31.861852    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:32.026524 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:32.026488    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:32.098768 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:32.098718    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8-registry-tls\") pod \"image-registry-9689ffdd5-8254l\" (UID: \"cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8\") " pod="openshift-image-registry/image-registry-9689ffdd5-8254l"
Sep 14 09:44:32.098768 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:32.098773    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4e19379f-15ef-494e-9760-8087d2a910ba-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-pdhmm\" (UID: \"4e19379f-15ef-494e-9760-8087d2a910ba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-pdhmm"
Sep 14 09:44:32.099070 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:32.098887    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 14 09:44:32.099070 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:32.098907    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9689ffdd5-8254l: secret "image-registry-tls" not found
Sep 14 09:44:32.099070 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:32.098930    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 14 09:44:32.099070 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:32.098959    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8-registry-tls podName:cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8 nodeName:}" failed. No retries permitted until 2026-09-14 09:44:33.098944523 +0000 UTC m=+62.138619122 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8-registry-tls") pod "image-registry-9689ffdd5-8254l" (UID: "cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8") : secret "image-registry-tls" not found
Sep 14 09:44:32.099070 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:32.098992    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e19379f-15ef-494e-9760-8087d2a910ba-samples-operator-tls podName:4e19379f-15ef-494e-9760-8087d2a910ba nodeName:}" failed. No retries permitted until 2026-09-14 09:44:33.098975994 +0000 UTC m=+62.138650594 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4e19379f-15ef-494e-9760-8087d2a910ba-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-pdhmm" (UID: "4e19379f-15ef-494e-9760-8087d2a910ba") : secret "samples-operator-tls" not found
Sep 14 09:44:32.602217 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:32.602168    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/161e5bed-e6f8-4c08-8c69-857064cc8c4f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-7tpzw\" (UID: \"161e5bed-e6f8-4c08-8c69-857064cc8c4f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-7tpzw"
Sep 14 09:44:32.602431 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:32.602355    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 14 09:44:32.602500 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:32.602443    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/161e5bed-e6f8-4c08-8c69-857064cc8c4f-cluster-monitoring-operator-tls podName:161e5bed-e6f8-4c08-8c69-857064cc8c4f nodeName:}" failed. No retries permitted until 2026-09-14 09:44:34.602420978 +0000 UTC m=+63.642095577 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/161e5bed-e6f8-4c08-8c69-857064cc8c4f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-7tpzw" (UID: "161e5bed-e6f8-4c08-8c69-857064cc8c4f") : secret "cluster-monitoring-operator-tls" not found
Sep 14 09:44:33.104975 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:33.104934    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8-registry-tls\") pod \"image-registry-9689ffdd5-8254l\" (UID: \"cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8\") " pod="openshift-image-registry/image-registry-9689ffdd5-8254l"
Sep 14 09:44:33.104975 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:33.104973    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4e19379f-15ef-494e-9760-8087d2a910ba-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-pdhmm\" (UID: \"4e19379f-15ef-494e-9760-8087d2a910ba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-pdhmm"
Sep 14 09:44:33.105412 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:33.105103    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 14 09:44:33.105412 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:33.105149    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e19379f-15ef-494e-9760-8087d2a910ba-samples-operator-tls podName:4e19379f-15ef-494e-9760-8087d2a910ba nodeName:}" failed. No retries permitted until 2026-09-14 09:44:35.105136824 +0000 UTC m=+64.144811424 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4e19379f-15ef-494e-9760-8087d2a910ba-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-pdhmm" (UID: "4e19379f-15ef-494e-9760-8087d2a910ba") : secret "samples-operator-tls" not found
Sep 14 09:44:33.105412 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:33.105221    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 14 09:44:33.105412 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:33.105242    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9689ffdd5-8254l: secret "image-registry-tls" not found
Sep 14 09:44:33.105412 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:33.105291    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8-registry-tls podName:cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8 nodeName:}" failed. No retries permitted until 2026-09-14 09:44:35.105276207 +0000 UTC m=+64.144950821 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8-registry-tls") pod "image-registry-9689ffdd5-8254l" (UID: "cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8") : secret "image-registry-tls" not found
Sep 14 09:44:34.615970 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:34.615927    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/161e5bed-e6f8-4c08-8c69-857064cc8c4f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-7tpzw\" (UID: \"161e5bed-e6f8-4c08-8c69-857064cc8c4f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-7tpzw"
Sep 14 09:44:34.616397 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:34.616086    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 14 09:44:34.616397 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:34.616152    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/161e5bed-e6f8-4c08-8c69-857064cc8c4f-cluster-monitoring-operator-tls podName:161e5bed-e6f8-4c08-8c69-857064cc8c4f nodeName:}" failed. No retries permitted until 2026-09-14 09:44:38.616138142 +0000 UTC m=+67.655812754 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/161e5bed-e6f8-4c08-8c69-857064cc8c4f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-7tpzw" (UID: "161e5bed-e6f8-4c08-8c69-857064cc8c4f") : secret "cluster-monitoring-operator-tls" not found
Sep 14 09:44:35.119792 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:35.119749    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8-registry-tls\") pod \"image-registry-9689ffdd5-8254l\" (UID: \"cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8\") " pod="openshift-image-registry/image-registry-9689ffdd5-8254l"
Sep 14 09:44:35.119792 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:35.119793    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4e19379f-15ef-494e-9760-8087d2a910ba-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-pdhmm\" (UID: \"4e19379f-15ef-494e-9760-8087d2a910ba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-pdhmm"
Sep 14 09:44:35.120021 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:35.119895    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 14 09:44:35.120021 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:35.119922    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 14 09:44:35.120021 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:35.119948    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9689ffdd5-8254l: secret "image-registry-tls" not found
Sep 14 09:44:35.120021 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:35.119948    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e19379f-15ef-494e-9760-8087d2a910ba-samples-operator-tls podName:4e19379f-15ef-494e-9760-8087d2a910ba nodeName:}" failed. No retries permitted until 2026-09-14 09:44:39.119935546 +0000 UTC m=+68.159610144 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4e19379f-15ef-494e-9760-8087d2a910ba-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-pdhmm" (UID: "4e19379f-15ef-494e-9760-8087d2a910ba") : secret "samples-operator-tls" not found
Sep 14 09:44:35.120178 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:35.120054    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8-registry-tls podName:cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8 nodeName:}" failed. No retries permitted until 2026-09-14 09:44:39.120026083 +0000 UTC m=+68.159700688 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8-registry-tls") pod "image-registry-9689ffdd5-8254l" (UID: "cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8") : secret "image-registry-tls" not found
Sep 14 09:44:35.355341 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:35.355295    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:35.363428 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:35.363385    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-rqpdl"]
Sep 14 09:44:35.394351 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:35.394264    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-rqpdl"]
Sep 14 09:44:35.394522 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:35.394407    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-rqpdl"
Sep 14 09:44:35.397172 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:35.397143    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 14 09:44:35.397562 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:35.397545    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 14 09:44:35.397650 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:35.397546    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-w9sh6\""
Sep 14 09:44:35.423048 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:35.422992    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jc4n\" (UniqueName: \"kubernetes.io/projected/f2e9bff6-a63c-4334-897a-1fe8d29ba23a-kube-api-access-7jc4n\") pod \"migrator-8b4d697c5-rqpdl\" (UID: \"f2e9bff6-a63c-4334-897a-1fe8d29ba23a\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-rqpdl"
Sep 14 09:44:35.450535 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:35.450503    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:35.523996 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:35.523960    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jc4n\" (UniqueName: \"kubernetes.io/projected/f2e9bff6-a63c-4334-897a-1fe8d29ba23a-kube-api-access-7jc4n\") pod \"migrator-8b4d697c5-rqpdl\" (UID: \"f2e9bff6-a63c-4334-897a-1fe8d29ba23a\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-rqpdl"
Sep 14 09:44:35.537721 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:35.537693    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jc4n\" (UniqueName: \"kubernetes.io/projected/f2e9bff6-a63c-4334-897a-1fe8d29ba23a-kube-api-access-7jc4n\") pod \"migrator-8b4d697c5-rqpdl\" (UID: \"f2e9bff6-a63c-4334-897a-1fe8d29ba23a\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-rqpdl"
Sep 14 09:44:35.555034 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:35.554991    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:35.597118 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:35.595147    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-tvs2c"]
Sep 14 09:44:35.615754 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:35.615716    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tvs2c"]
Sep 14 09:44:35.615961 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:35.615869    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tvs2c"
Sep 14 09:44:35.618038 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:35.617962    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 14 09:44:35.618038 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:35.617962    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 14 09:44:35.618038 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:35.617973    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 14 09:44:35.618534 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:35.618289    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 14 09:44:35.618534 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:35.618343    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-22v7l\""
Sep 14 09:44:35.624698 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:35.624669    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps7gz\" (UniqueName: \"kubernetes.io/projected/bf286afe-8eb2-4e87-bc9c-5d8646662e58-kube-api-access-ps7gz\") pod \"insights-runtime-extractor-tvs2c\" (UID: \"bf286afe-8eb2-4e87-bc9c-5d8646662e58\") " pod="openshift-insights/insights-runtime-extractor-tvs2c"
Sep 14 09:44:35.624842 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:35.624729    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bf286afe-8eb2-4e87-bc9c-5d8646662e58-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tvs2c\" (UID: \"bf286afe-8eb2-4e87-bc9c-5d8646662e58\") " pod="openshift-insights/insights-runtime-extractor-tvs2c"
Sep 14 09:44:35.624911 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:35.624840    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bf286afe-8eb2-4e87-bc9c-5d8646662e58-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tvs2c\" (UID: \"bf286afe-8eb2-4e87-bc9c-5d8646662e58\") " pod="openshift-insights/insights-runtime-extractor-tvs2c"
Sep 14 09:44:35.624911 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:35.624870    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bf286afe-8eb2-4e87-bc9c-5d8646662e58-data-volume\") pod \"insights-runtime-extractor-tvs2c\" (UID: \"bf286afe-8eb2-4e87-bc9c-5d8646662e58\") " pod="openshift-insights/insights-runtime-extractor-tvs2c"
Sep 14 09:44:35.624911 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:35.624900    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bf286afe-8eb2-4e87-bc9c-5d8646662e58-crio-socket\") pod \"insights-runtime-extractor-tvs2c\" (UID: \"bf286afe-8eb2-4e87-bc9c-5d8646662e58\") " pod="openshift-insights/insights-runtime-extractor-tvs2c"
Sep 14 09:44:35.650033 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:35.649923    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:35.704697 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:35.704661    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-rqpdl"
Sep 14 09:44:35.726158 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:35.726128    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps7gz\" (UniqueName: \"kubernetes.io/projected/bf286afe-8eb2-4e87-bc9c-5d8646662e58-kube-api-access-ps7gz\") pod \"insights-runtime-extractor-tvs2c\" (UID: \"bf286afe-8eb2-4e87-bc9c-5d8646662e58\") " pod="openshift-insights/insights-runtime-extractor-tvs2c"
Sep 14 09:44:35.726297 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:35.726192    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bf286afe-8eb2-4e87-bc9c-5d8646662e58-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tvs2c\" (UID: \"bf286afe-8eb2-4e87-bc9c-5d8646662e58\") " pod="openshift-insights/insights-runtime-extractor-tvs2c"
Sep 14 09:44:35.726297 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:35.726268    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bf286afe-8eb2-4e87-bc9c-5d8646662e58-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tvs2c\" (UID: \"bf286afe-8eb2-4e87-bc9c-5d8646662e58\") " pod="openshift-insights/insights-runtime-extractor-tvs2c"
Sep 14 09:44:35.726365 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:35.726302    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bf286afe-8eb2-4e87-bc9c-5d8646662e58-data-volume\") pod \"insights-runtime-extractor-tvs2c\" (UID: \"bf286afe-8eb2-4e87-bc9c-5d8646662e58\") " pod="openshift-insights/insights-runtime-extractor-tvs2c"
Sep 14 09:44:35.726365 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:35.726336    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bf286afe-8eb2-4e87-bc9c-5d8646662e58-crio-socket\") pod \"insights-runtime-extractor-tvs2c\" (UID: \"bf286afe-8eb2-4e87-bc9c-5d8646662e58\") " pod="openshift-insights/insights-runtime-extractor-tvs2c"
Sep 14 09:44:35.726431 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:35.726377    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 14 09:44:35.726474 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:35.726432    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 14 09:44:35.726474 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:35.726458    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bf286afe-8eb2-4e87-bc9c-5d8646662e58-crio-socket\") pod \"insights-runtime-extractor-tvs2c\" (UID: \"bf286afe-8eb2-4e87-bc9c-5d8646662e58\") " pod="openshift-insights/insights-runtime-extractor-tvs2c"
Sep 14 09:44:35.726550 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:35.726460    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bf286afe-8eb2-4e87-bc9c-5d8646662e58-kube-rbac-proxy-cm podName:bf286afe-8eb2-4e87-bc9c-5d8646662e58 nodeName:}" failed. No retries permitted until 2026-09-14 09:44:36.226437813 +0000 UTC m=+65.266112436 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bf286afe-8eb2-4e87-bc9c-5d8646662e58-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tvs2c" (UID: "bf286afe-8eb2-4e87-bc9c-5d8646662e58") : configmap "kube-rbac-proxy" not found
Sep 14 09:44:35.726550 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:35.726502    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf286afe-8eb2-4e87-bc9c-5d8646662e58-insights-runtime-extractor-tls podName:bf286afe-8eb2-4e87-bc9c-5d8646662e58 nodeName:}" failed. No retries permitted until 2026-09-14 09:44:36.226486794 +0000 UTC m=+65.266161392 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bf286afe-8eb2-4e87-bc9c-5d8646662e58-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tvs2c" (UID: "bf286afe-8eb2-4e87-bc9c-5d8646662e58") : secret "insights-runtime-extractor-tls" not found
Sep 14 09:44:35.726672 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:35.726653    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bf286afe-8eb2-4e87-bc9c-5d8646662e58-data-volume\") pod \"insights-runtime-extractor-tvs2c\" (UID: \"bf286afe-8eb2-4e87-bc9c-5d8646662e58\") " pod="openshift-insights/insights-runtime-extractor-tvs2c"
Sep 14 09:44:35.734571 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:35.734544    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps7gz\" (UniqueName: \"kubernetes.io/projected/bf286afe-8eb2-4e87-bc9c-5d8646662e58-kube-api-access-ps7gz\") pod \"insights-runtime-extractor-tvs2c\" (UID: \"bf286afe-8eb2-4e87-bc9c-5d8646662e58\") " pod="openshift-insights/insights-runtime-extractor-tvs2c"
Sep 14 09:44:36.001473 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:36.001437    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-rqpdl"]
Sep 14 09:44:36.004965 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:44:36.004931    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2e9bff6_a63c_4334_897a_1fe8d29ba23a.slice/crio-8b7fca87e316d0fabbaa1330f0470c6c2d6c35ebfaaa3305aee13cd645d386ab WatchSource:0}: Error finding container 8b7fca87e316d0fabbaa1330f0470c6c2d6c35ebfaaa3305aee13cd645d386ab: Status 404 returned error can't find the container with id 8b7fca87e316d0fabbaa1330f0470c6c2d6c35ebfaaa3305aee13cd645d386ab
Sep 14 09:44:36.057394 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:36.057362    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:36.142424 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:36.142388    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:36.231161 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:36.231115    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bf286afe-8eb2-4e87-bc9c-5d8646662e58-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tvs2c\" (UID: \"bf286afe-8eb2-4e87-bc9c-5d8646662e58\") " pod="openshift-insights/insights-runtime-extractor-tvs2c"
Sep 14 09:44:36.231328 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:36.231186    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bf286afe-8eb2-4e87-bc9c-5d8646662e58-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tvs2c\" (UID: \"bf286afe-8eb2-4e87-bc9c-5d8646662e58\") " pod="openshift-insights/insights-runtime-extractor-tvs2c"
Sep 14 09:44:36.231328 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:36.231243    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 14 09:44:36.231328 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:36.231317    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bf286afe-8eb2-4e87-bc9c-5d8646662e58-kube-rbac-proxy-cm podName:bf286afe-8eb2-4e87-bc9c-5d8646662e58 nodeName:}" failed. No retries permitted until 2026-09-14 09:44:37.231297701 +0000 UTC m=+66.270972322 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bf286afe-8eb2-4e87-bc9c-5d8646662e58-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tvs2c" (UID: "bf286afe-8eb2-4e87-bc9c-5d8646662e58") : configmap "kube-rbac-proxy" not found
Sep 14 09:44:36.231432 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:36.231332    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 14 09:44:36.231432 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:36.231383    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf286afe-8eb2-4e87-bc9c-5d8646662e58-insights-runtime-extractor-tls podName:bf286afe-8eb2-4e87-bc9c-5d8646662e58 nodeName:}" failed. No retries permitted until 2026-09-14 09:44:37.2313694 +0000 UTC m=+66.271044016 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bf286afe-8eb2-4e87-bc9c-5d8646662e58-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tvs2c" (UID: "bf286afe-8eb2-4e87-bc9c-5d8646662e58") : secret "insights-runtime-extractor-tls" not found
Sep 14 09:44:36.353026 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:36.352904    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:36.453733 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:36.453698    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:36.878692 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:36.878646    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-rqpdl" event={"ID":"f2e9bff6-a63c-4334-897a-1fe8d29ba23a","Type":"ContainerStarted","Data":"8b7fca87e316d0fabbaa1330f0470c6c2d6c35ebfaaa3305aee13cd645d386ab"}
Sep 14 09:44:37.238987 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:37.238951    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bf286afe-8eb2-4e87-bc9c-5d8646662e58-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tvs2c\" (UID: \"bf286afe-8eb2-4e87-bc9c-5d8646662e58\") " pod="openshift-insights/insights-runtime-extractor-tvs2c"
Sep 14 09:44:37.239148 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:37.239020    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d956c018-b9cb-4c19-8f08-16652611d8c3-metrics-certs\") pod \"network-metrics-daemon-6l89k\" (UID: \"d956c018-b9cb-4c19-8f08-16652611d8c3\") " pod="openshift-multus/network-metrics-daemon-6l89k"
Sep 14 09:44:37.239148 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:37.239060    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97284aa2-9a80-495f-8b99-2ff7cec913da-original-pull-secret\") pod \"global-pull-secret-syncer-krg4v\" (UID: \"97284aa2-9a80-495f-8b99-2ff7cec913da\") " pod="kube-system/global-pull-secret-syncer-krg4v"
Sep 14 09:44:37.239148 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:37.239109    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 14 09:44:37.239289 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:37.239176    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bf286afe-8eb2-4e87-bc9c-5d8646662e58-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tvs2c\" (UID: \"bf286afe-8eb2-4e87-bc9c-5d8646662e58\") " pod="openshift-insights/insights-runtime-extractor-tvs2c"
Sep 14 09:44:37.239289 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:37.239189    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bf286afe-8eb2-4e87-bc9c-5d8646662e58-kube-rbac-proxy-cm podName:bf286afe-8eb2-4e87-bc9c-5d8646662e58 nodeName:}" failed. No retries permitted until 2026-09-14 09:44:39.239168113 +0000 UTC m=+68.278842727 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bf286afe-8eb2-4e87-bc9c-5d8646662e58-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tvs2c" (UID: "bf286afe-8eb2-4e87-bc9c-5d8646662e58") : configmap "kube-rbac-proxy" not found
Sep 14 09:44:37.239289 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:37.239245    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 14 09:44:37.239428 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:37.239310    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf286afe-8eb2-4e87-bc9c-5d8646662e58-insights-runtime-extractor-tls podName:bf286afe-8eb2-4e87-bc9c-5d8646662e58 nodeName:}" failed. No retries permitted until 2026-09-14 09:44:39.239294041 +0000 UTC m=+68.278968652 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bf286afe-8eb2-4e87-bc9c-5d8646662e58-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tvs2c" (UID: "bf286afe-8eb2-4e87-bc9c-5d8646662e58") : secret "insights-runtime-extractor-tls" not found
Sep 14 09:44:37.242822 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:37.242532    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 14 09:44:37.243260 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:37.243238    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 14 09:44:37.250064 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:37.250033    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 14 09:44:37.250229 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:37.250117    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d956c018-b9cb-4c19-8f08-16652611d8c3-metrics-certs podName:d956c018-b9cb-4c19-8f08-16652611d8c3 nodeName:}" failed. No retries permitted until 2026-09-14 09:45:41.250096468 +0000 UTC m=+130.289771082 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d956c018-b9cb-4c19-8f08-16652611d8c3-metrics-certs") pod "network-metrics-daemon-6l89k" (UID: "d956c018-b9cb-4c19-8f08-16652611d8c3") : secret "metrics-daemon-secret" not found
Sep 14 09:44:37.252244 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:37.252219    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97284aa2-9a80-495f-8b99-2ff7cec913da-original-pull-secret\") pod \"global-pull-secret-syncer-krg4v\" (UID: \"97284aa2-9a80-495f-8b99-2ff7cec913da\") " pod="kube-system/global-pull-secret-syncer-krg4v"
Sep 14 09:44:37.340547 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:37.340502    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkm5l\" (UniqueName: \"kubernetes.io/projected/c41422ae-06c2-492c-9dbf-82e10222589a-kube-api-access-xkm5l\") pod \"network-check-target-8zcts\" (UID: \"c41422ae-06c2-492c-9dbf-82e10222589a\") " pod="openshift-network-diagnostics/network-check-target-8zcts"
Sep 14 09:44:37.343081 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:37.343054    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 14 09:44:37.353422 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:37.353389    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 14 09:44:37.363984 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:37.363950    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkm5l\" (UniqueName: \"kubernetes.io/projected/c41422ae-06c2-492c-9dbf-82e10222589a-kube-api-access-xkm5l\") pod \"network-check-target-8zcts\" (UID: \"c41422ae-06c2-492c-9dbf-82e10222589a\") " pod="openshift-network-diagnostics/network-check-target-8zcts"
Sep 14 09:44:37.390280 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:37.390243    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-clrv6\""
Sep 14 09:44:37.398258 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:37.398214    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8zcts"
Sep 14 09:44:37.399492 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:37.399464    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krg4v"
Sep 14 09:44:37.642344 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:37.642307    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec1015e1-7070-4f02-8f4e-2fd5d395c4e9-metrics-tls\") pod \"dns-default-mklrw\" (UID: \"ec1015e1-7070-4f02-8f4e-2fd5d395c4e9\") " pod="openshift-dns/dns-default-mklrw"
Sep 14 09:44:37.642529 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:37.642441    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 14 09:44:37.642529 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:37.642495    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec1015e1-7070-4f02-8f4e-2fd5d395c4e9-metrics-tls podName:ec1015e1-7070-4f02-8f4e-2fd5d395c4e9 nodeName:}" failed. No retries permitted until 2026-09-14 09:45:09.642481707 +0000 UTC m=+98.682156307 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ec1015e1-7070-4f02-8f4e-2fd5d395c4e9-metrics-tls") pod "dns-default-mklrw" (UID: "ec1015e1-7070-4f02-8f4e-2fd5d395c4e9") : secret "dns-default-metrics-tls" not found
Sep 14 09:44:37.680636 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:37.680609    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8zcts"]
Sep 14 09:44:37.682773 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:44:37.682737    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc41422ae_06c2_492c_9dbf_82e10222589a.slice/crio-c87ff31ae1689edb21571a5bbbb57a8171e56f27f7c30c09c9fcc7a8bba3b63f WatchSource:0}: Error finding container c87ff31ae1689edb21571a5bbbb57a8171e56f27f7c30c09c9fcc7a8bba3b63f: Status 404 returned error can't find the container with id c87ff31ae1689edb21571a5bbbb57a8171e56f27f7c30c09c9fcc7a8bba3b63f
Sep 14 09:44:37.683596 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:37.683524    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-krg4v"]
Sep 14 09:44:37.687552 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:44:37.687521    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97284aa2_9a80_495f_8b99_2ff7cec913da.slice/crio-3a3b9214a69ff6e6848a081e641709c211d35cc5afe539d45cd6f840f031dcf7 WatchSource:0}: Error finding container 3a3b9214a69ff6e6848a081e641709c211d35cc5afe539d45cd6f840f031dcf7: Status 404 returned error can't find the container with id 3a3b9214a69ff6e6848a081e641709c211d35cc5afe539d45cd6f840f031dcf7
Sep 14 09:44:37.882971 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:37.882926    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-rqpdl" event={"ID":"f2e9bff6-a63c-4334-897a-1fe8d29ba23a","Type":"ContainerStarted","Data":"97fcd604cd06a19676c2e6902bbef467e9bbd0bfcc29b627c75a35389d9cfbe1"}
Sep 14 09:44:37.883400 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:37.882988    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-rqpdl" event={"ID":"f2e9bff6-a63c-4334-897a-1fe8d29ba23a","Type":"ContainerStarted","Data":"07a41ae25d4c974b339c0447ca739f389bf03d141d4d90e61e0ee37d27ca94e7"}
Sep 14 09:44:37.884045 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:37.884021    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-krg4v" event={"ID":"97284aa2-9a80-495f-8b99-2ff7cec913da","Type":"ContainerStarted","Data":"3a3b9214a69ff6e6848a081e641709c211d35cc5afe539d45cd6f840f031dcf7"}
Sep 14 09:44:37.885139 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:37.885119    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8zcts" event={"ID":"c41422ae-06c2-492c-9dbf-82e10222589a","Type":"ContainerStarted","Data":"c87ff31ae1689edb21571a5bbbb57a8171e56f27f7c30c09c9fcc7a8bba3b63f"}
Sep 14 09:44:37.920122 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:37.920068    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-rqpdl" podStartSLOduration=1.7689147360000002 podStartE2EDuration="2.920040648s" totalImagesPullingTime="1.151125912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:44:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:44:37.920033555 +0000 UTC m=+66.959708176" watchObservedRunningTime="2026-09-14 09:44:37.920040648 +0000 UTC m=+66.959715279"
Sep 14 09:44:38.059414 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:38.059377    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:38.214396 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:38.214357    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:38.333671 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:38.333594    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-f5jts_da4b31da-fcbb-4132-bc49-1ecd91bb65bb/dns-node-resolver/0.log"
Sep 14 09:44:38.651440 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:38.651338    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/161e5bed-e6f8-4c08-8c69-857064cc8c4f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-7tpzw\" (UID: \"161e5bed-e6f8-4c08-8c69-857064cc8c4f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-7tpzw"
Sep 14 09:44:38.651617 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:38.651501    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 14 09:44:38.651617 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:38.651581    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/161e5bed-e6f8-4c08-8c69-857064cc8c4f-cluster-monitoring-operator-tls podName:161e5bed-e6f8-4c08-8c69-857064cc8c4f nodeName:}" failed. No retries permitted until 2026-09-14 09:44:46.651560179 +0000 UTC m=+75.691234797 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/161e5bed-e6f8-4c08-8c69-857064cc8c4f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-7tpzw" (UID: "161e5bed-e6f8-4c08-8c69-857064cc8c4f") : secret "cluster-monitoring-operator-tls" not found
Sep 14 09:44:39.156677 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:39.156624    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8-registry-tls\") pod \"image-registry-9689ffdd5-8254l\" (UID: \"cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8\") " pod="openshift-image-registry/image-registry-9689ffdd5-8254l"
Sep 14 09:44:39.156677 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:39.156682    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4e19379f-15ef-494e-9760-8087d2a910ba-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-pdhmm\" (UID: \"4e19379f-15ef-494e-9760-8087d2a910ba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-pdhmm"
Sep 14 09:44:39.157221 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:39.156817    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 14 09:44:39.157221 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:39.156842    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9689ffdd5-8254l: secret "image-registry-tls" not found
Sep 14 09:44:39.157221 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:39.156907    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 14 09:44:39.157221 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:39.156914    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8-registry-tls podName:cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8 nodeName:}" failed. No retries permitted until 2026-09-14 09:44:47.156889881 +0000 UTC m=+76.196564492 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8-registry-tls") pod "image-registry-9689ffdd5-8254l" (UID: "cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8") : secret "image-registry-tls" not found
Sep 14 09:44:39.157221 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:39.156961    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e19379f-15ef-494e-9760-8087d2a910ba-samples-operator-tls podName:4e19379f-15ef-494e-9760-8087d2a910ba nodeName:}" failed. No retries permitted until 2026-09-14 09:44:47.15694469 +0000 UTC m=+76.196619303 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4e19379f-15ef-494e-9760-8087d2a910ba-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-pdhmm" (UID: "4e19379f-15ef-494e-9760-8087d2a910ba") : secret "samples-operator-tls" not found
Sep 14 09:44:39.257710 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:39.257666    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bf286afe-8eb2-4e87-bc9c-5d8646662e58-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tvs2c\" (UID: \"bf286afe-8eb2-4e87-bc9c-5d8646662e58\") " pod="openshift-insights/insights-runtime-extractor-tvs2c"
Sep 14 09:44:39.257902 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:39.257827    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 14 09:44:39.257902 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:39.257855    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bf286afe-8eb2-4e87-bc9c-5d8646662e58-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tvs2c\" (UID: \"bf286afe-8eb2-4e87-bc9c-5d8646662e58\") " pod="openshift-insights/insights-runtime-extractor-tvs2c"
Sep 14 09:44:39.258012 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:39.257904    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bf286afe-8eb2-4e87-bc9c-5d8646662e58-kube-rbac-proxy-cm podName:bf286afe-8eb2-4e87-bc9c-5d8646662e58 nodeName:}" failed. No retries permitted until 2026-09-14 09:44:43.257886832 +0000 UTC m=+72.297561430 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bf286afe-8eb2-4e87-bc9c-5d8646662e58-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tvs2c" (UID: "bf286afe-8eb2-4e87-bc9c-5d8646662e58") : configmap "kube-rbac-proxy" not found
Sep 14 09:44:39.258012 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:39.257980    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 14 09:44:39.258112 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:39.258055    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf286afe-8eb2-4e87-bc9c-5d8646662e58-insights-runtime-extractor-tls podName:bf286afe-8eb2-4e87-bc9c-5d8646662e58 nodeName:}" failed. No retries permitted until 2026-09-14 09:44:43.258038131 +0000 UTC m=+72.297712749 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bf286afe-8eb2-4e87-bc9c-5d8646662e58-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tvs2c" (UID: "bf286afe-8eb2-4e87-bc9c-5d8646662e58") : secret "insights-runtime-extractor-tls" not found
Sep 14 09:44:39.330623 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:39.330594    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mq9n4_07d5e1df-7d96-401d-b606-245e4b1f3ae5/node-ca/0.log"
Sep 14 09:44:39.424334 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:39.424225    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:40.131194 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:40.131084    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-f7d9v_4a0bee38-c70e-420f-82f8-ff4f0c42e0a6/serve-healthcheck-canary/0.log"
Sep 14 09:44:40.222576 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:40.222537    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:40.321015 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:40.320955    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:40.523221 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:40.523175    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:40.531389 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:40.531359    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-rqpdl_f2e9bff6-a63c-4334-897a-1fe8d29ba23a/migrator/0.log"
Sep 14 09:44:40.737487 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:40.737459    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-rqpdl_f2e9bff6-a63c-4334-897a-1fe8d29ba23a/graceful-termination/0.log"
Sep 14 09:44:40.773126 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:40.773087    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:40.882765 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:40.882671    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:40.982462 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:40.982419    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:41.722952 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:41.722910    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:42.122823 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:42.122788    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:42.568670 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:42.568628    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:42.900429 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:42.900390    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-krg4v" event={"ID":"97284aa2-9a80-495f-8b99-2ff7cec913da","Type":"ContainerStarted","Data":"571cee84bfce74248d1af671da1e5cce986bb01a599f2839e1064af535add49a"}
Sep 14 09:44:42.902110 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:42.902084    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8zcts" event={"ID":"c41422ae-06c2-492c-9dbf-82e10222589a","Type":"ContainerStarted","Data":"f75be1168024b6d762aa0260cbac1f075eedbe297bbdcc2d5963aa019339f424"}
Sep 14 09:44:42.902257 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:42.902148    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8zcts"
Sep 14 09:44:42.943206 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:42.943152    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-krg4v" podStartSLOduration=67.353818607 podStartE2EDuration="1m11.943133654s" totalImagesPullingTime="4.589315047s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 09:43:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:44:42.94188771 +0000 UTC m=+71.981562331" watchObservedRunningTime="2026-09-14 09:44:42.943133654 +0000 UTC m=+71.982808276"
Sep 14 09:44:42.964122 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:42.964053    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8zcts" podStartSLOduration=67.374075082 podStartE2EDuration="1m11.964033624s" totalImagesPullingTime="4.589958542s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:43:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:44:42.96370791 +0000 UTC m=+72.003382533" watchObservedRunningTime="2026-09-14 09:44:42.964033624 +0000 UTC m=+72.003708242"
Sep 14 09:44:42.970388 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:42.970347    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:43.184672 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:43.184553    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:43.295896 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:43.295848    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bf286afe-8eb2-4e87-bc9c-5d8646662e58-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tvs2c\" (UID: \"bf286afe-8eb2-4e87-bc9c-5d8646662e58\") " pod="openshift-insights/insights-runtime-extractor-tvs2c"
Sep 14 09:44:43.296102 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:43.295920    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bf286afe-8eb2-4e87-bc9c-5d8646662e58-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tvs2c\" (UID: \"bf286afe-8eb2-4e87-bc9c-5d8646662e58\") " pod="openshift-insights/insights-runtime-extractor-tvs2c"
Sep 14 09:44:43.296102 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:43.295981    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 14 09:44:43.296102 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:43.296093    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bf286afe-8eb2-4e87-bc9c-5d8646662e58-kube-rbac-proxy-cm podName:bf286afe-8eb2-4e87-bc9c-5d8646662e58 nodeName:}" failed. No retries permitted until 2026-09-14 09:44:51.296071009 +0000 UTC m=+80.335745611 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bf286afe-8eb2-4e87-bc9c-5d8646662e58-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tvs2c" (UID: "bf286afe-8eb2-4e87-bc9c-5d8646662e58") : configmap "kube-rbac-proxy" not found
Sep 14 09:44:43.296219 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:43.296129    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 14 09:44:43.296219 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:43.296184    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf286afe-8eb2-4e87-bc9c-5d8646662e58-insights-runtime-extractor-tls podName:bf286afe-8eb2-4e87-bc9c-5d8646662e58 nodeName:}" failed. No retries permitted until 2026-09-14 09:44:51.296170995 +0000 UTC m=+80.335845603 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bf286afe-8eb2-4e87-bc9c-5d8646662e58-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tvs2c" (UID: "bf286afe-8eb2-4e87-bc9c-5d8646662e58") : secret "insights-runtime-extractor-tls" not found
Sep 14 09:44:43.333541 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:43.333507    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:43.907062 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:43.906988    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:44.173720 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:44.173627    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:44.272851 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:44.272817    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:44.734244 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:44.734199    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:44.936168 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:44.936131    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:45.035060 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:45.034949    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:45.131815 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:45.131782    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:45.517130 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:45.517095    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:46.309299 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:46.309265    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:46.523422 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:46.523393    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:46.600403 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:46.600311    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:46.723909 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:46.723871    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/161e5bed-e6f8-4c08-8c69-857064cc8c4f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-7tpzw\" (UID: \"161e5bed-e6f8-4c08-8c69-857064cc8c4f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-7tpzw"
Sep 14 09:44:46.724057 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:46.723992    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 14 09:44:46.724108 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:44:46.724067    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/161e5bed-e6f8-4c08-8c69-857064cc8c4f-cluster-monitoring-operator-tls podName:161e5bed-e6f8-4c08-8c69-857064cc8c4f nodeName:}" failed. No retries permitted until 2026-09-14 09:45:02.724053203 +0000 UTC m=+91.763727802 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/161e5bed-e6f8-4c08-8c69-857064cc8c4f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-7tpzw" (UID: "161e5bed-e6f8-4c08-8c69-857064cc8c4f") : secret "cluster-monitoring-operator-tls" not found
Sep 14 09:44:46.917626 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:46.917536    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:47.106628 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:47.106596    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:47.227126 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:47.227054    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8-registry-tls\") pod \"image-registry-9689ffdd5-8254l\" (UID: \"cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8\") " pod="openshift-image-registry/image-registry-9689ffdd5-8254l"
Sep 14 09:44:47.227126 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:47.227094    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4e19379f-15ef-494e-9760-8087d2a910ba-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-pdhmm\" (UID: \"4e19379f-15ef-494e-9760-8087d2a910ba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-pdhmm"
Sep 14 09:44:47.229630 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:47.229592    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8-registry-tls\") pod \"image-registry-9689ffdd5-8254l\" (UID: \"cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8\") " pod="openshift-image-registry/image-registry-9689ffdd5-8254l"
Sep 14 09:44:47.229630 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:47.229627    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4e19379f-15ef-494e-9760-8087d2a910ba-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-pdhmm\" (UID: \"4e19379f-15ef-494e-9760-8087d2a910ba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-pdhmm"
Sep 14 09:44:47.340257 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:47.340218    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-9689ffdd5-8254l"
Sep 14 09:44:47.359422 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:47.359383    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-pdhmm"
Sep 14 09:44:47.508762 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:47.508722    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:47.618417 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:47.618382    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-9689ffdd5-8254l"]
Sep 14 09:44:47.621567 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:44:47.621535    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd1d6690_5f55_428e_b4e2_64c3a2a7b4d8.slice/crio-c96b1e4306b0fe1fa6b320205ed92126a13ce01819d4f230298c483616ee7dcb WatchSource:0}: Error finding container c96b1e4306b0fe1fa6b320205ed92126a13ce01819d4f230298c483616ee7dcb: Status 404 returned error can't find the container with id c96b1e4306b0fe1fa6b320205ed92126a13ce01819d4f230298c483616ee7dcb
Sep 14 09:44:47.641761 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:47.641725    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-pdhmm"]
Sep 14 09:44:47.917875 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:47.917769    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-9689ffdd5-8254l" event={"ID":"cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8","Type":"ContainerStarted","Data":"5060c5e4d3132cbe99409e76608c33b243b952cb3c684e4c06b665fd117b912d"}
Sep 14 09:44:47.917875 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:47.917827    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-9689ffdd5-8254l" event={"ID":"cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8","Type":"ContainerStarted","Data":"c96b1e4306b0fe1fa6b320205ed92126a13ce01819d4f230298c483616ee7dcb"}
Sep 14 09:44:47.918343 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:47.918314    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-9689ffdd5-8254l"
Sep 14 09:44:47.923523 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:47.923490    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-pdhmm" event={"ID":"4e19379f-15ef-494e-9760-8087d2a910ba","Type":"ContainerStarted","Data":"67c978fcd17812683fbcf8168fa1e31cc18a65223abd9c365b35f3a235f4684e"}
Sep 14 09:44:47.941424 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:47.941364    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-9689ffdd5-8254l" podStartSLOduration=16.941347405 podStartE2EDuration="16.941347405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:44:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:44:47.940468383 +0000 UTC m=+76.980143004" watchObservedRunningTime="2026-09-14 09:44:47.941347405 +0000 UTC m=+76.981022024"
Sep 14 09:44:48.351546 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:48.351499    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:48.562124 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:48.562092    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:48.920568 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:48.920528    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:49.135476 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:49.135420    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:49.516042 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:49.515989    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:49.932069 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:49.932030    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-pdhmm" event={"ID":"4e19379f-15ef-494e-9760-8087d2a910ba","Type":"ContainerStarted","Data":"5498327bd6a7fd251c4db7ea8d628a9d33e8667aa3668da593e813fb659122e4"}
Sep 14 09:44:49.932264 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:49.932082    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-pdhmm" event={"ID":"4e19379f-15ef-494e-9760-8087d2a910ba","Type":"ContainerStarted","Data":"006fdfd8e696cd612ddce6e271679039713c55c059fc85ceeda75903bddbcaa6"}
Sep 14 09:44:50.344643 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:50.344603    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:50.530133 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:50.530089    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:50.758632 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:50.758598    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:50.933577 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:50.933545    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:51.362561 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:51.362532    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bf286afe-8eb2-4e87-bc9c-5d8646662e58-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tvs2c\" (UID: \"bf286afe-8eb2-4e87-bc9c-5d8646662e58\") " pod="openshift-insights/insights-runtime-extractor-tvs2c"
Sep 14 09:44:51.362716 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:51.362587    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bf286afe-8eb2-4e87-bc9c-5d8646662e58-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tvs2c\" (UID: \"bf286afe-8eb2-4e87-bc9c-5d8646662e58\") " pod="openshift-insights/insights-runtime-extractor-tvs2c"
Sep 14 09:44:51.363094 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:51.363063    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bf286afe-8eb2-4e87-bc9c-5d8646662e58-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tvs2c\" (UID: \"bf286afe-8eb2-4e87-bc9c-5d8646662e58\") " pod="openshift-insights/insights-runtime-extractor-tvs2c"
Sep 14 09:44:51.364929 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:51.364905    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bf286afe-8eb2-4e87-bc9c-5d8646662e58-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tvs2c\" (UID: \"bf286afe-8eb2-4e87-bc9c-5d8646662e58\") " pod="openshift-insights/insights-runtime-extractor-tvs2c"
Sep 14 09:44:51.525531 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:51.525488    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tvs2c"
Sep 14 09:44:51.807934 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:51.807859    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-pdhmm" podStartSLOduration=19.169514489 podStartE2EDuration="20.80784061s" totalImagesPullingTime="1.638326121s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:44:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:44:49.9573842 +0000 UTC m=+78.997058823" watchObservedRunningTime="2026-09-14 09:44:51.80784061 +0000 UTC m=+80.847515230"
Sep 14 09:44:51.808431 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:51.808413    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tvs2c"]
Sep 14 09:44:51.812454 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:44:51.812421    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf286afe_8eb2_4e87_bc9c_5d8646662e58.slice/crio-0f7a211dea83abead8903c302f5f3a60a4b0b4e0e84a9e79d4b372dc8d1277e8 WatchSource:0}: Error finding container 0f7a211dea83abead8903c302f5f3a60a4b0b4e0e84a9e79d4b372dc8d1277e8: Status 404 returned error can't find the container with id 0f7a211dea83abead8903c302f5f3a60a4b0b4e0e84a9e79d4b372dc8d1277e8
Sep 14 09:44:51.940233 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:51.940188    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tvs2c" event={"ID":"bf286afe-8eb2-4e87-bc9c-5d8646662e58","Type":"ContainerStarted","Data":"cd8ffe83d55cda3001a1d18113a9cfaebfaece47f1f962aa0a41575ac18de22a"}
Sep 14 09:44:51.940403 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:51.940258    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tvs2c" event={"ID":"bf286afe-8eb2-4e87-bc9c-5d8646662e58","Type":"ContainerStarted","Data":"0f7a211dea83abead8903c302f5f3a60a4b0b4e0e84a9e79d4b372dc8d1277e8"}
Sep 14 09:44:51.993485 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:51.993446    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:52.945575 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:52.945480    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tvs2c" event={"ID":"bf286afe-8eb2-4e87-bc9c-5d8646662e58","Type":"ContainerStarted","Data":"e8114cc6f98fb3ed2b614e1738d2573a04f59bf4b9a7375ffa9b4eee2d7ef9e8"}
Sep 14 09:44:54.958586 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:54.958550    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tvs2c" event={"ID":"bf286afe-8eb2-4e87-bc9c-5d8646662e58","Type":"ContainerStarted","Data":"03e66ee1a69ca40c208fbb2e729726cff9fe9fb953cf5ce48772063784b96a0e"}
Sep 14 09:44:54.981180 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:54.981122    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-tvs2c" podStartSLOduration=17.91701782 podStartE2EDuration="19.981104845s" totalImagesPullingTime="2.064087025s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 09:44:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:44:54.980184802 +0000 UTC m=+84.019859420" watchObservedRunningTime="2026-09-14 09:44:54.981104845 +0000 UTC m=+84.020779466"
Sep 14 09:44:56.652934 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:56.652899    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:56.754669 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:56.754631    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:56.853711 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:56.853678    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:56.958705 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:56.958616    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:57.118016 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:57.117951    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:57.197244 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:57.197206    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:57.512161 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:57.512127    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:57.579641 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:57.579598    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:57.920747 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:57.920650    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:58.032469 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:58.032430    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:58.354197 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:58.354160    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:58.745634 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:58.745597    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:59.443202 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:59.443171    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:59.551203 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:59.551171    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:44:59.633086 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:44:59.633049    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:00.372117 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:00.372079    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79c94f89b6-6stck"]
Sep 14 09:45:00.376346 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:00.376321    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79c94f89b6-6stck"
Sep 14 09:45:00.379545 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:00.379521    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 14 09:45:00.379929 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:00.379913    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 14 09:45:00.385139 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:00.385108    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 14 09:45:00.385594 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:00.385580    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-78tlg\""
Sep 14 09:45:00.391018 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:00.390816    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 14 09:45:00.394109 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:00.392649    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79c94f89b6-6stck"]
Sep 14 09:45:00.411603 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:00.411566    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-9689ffdd5-8254l"]
Sep 14 09:45:00.434074 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:00.434028    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2ed75443-bd9e-40ad-8a4f-b9047cce9f53-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-79c94f89b6-6stck\" (UID: \"2ed75443-bd9e-40ad-8a4f-b9047cce9f53\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79c94f89b6-6stck"
Sep 14 09:45:00.434268 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:00.434097    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7zdq\" (UniqueName: \"kubernetes.io/projected/2ed75443-bd9e-40ad-8a4f-b9047cce9f53-kube-api-access-f7zdq\") pod \"managed-serviceaccount-addon-agent-79c94f89b6-6stck\" (UID: \"2ed75443-bd9e-40ad-8a4f-b9047cce9f53\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79c94f89b6-6stck"
Sep 14 09:45:00.512630 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:00.512591    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-544f748fdb-5pr6x"]
Sep 14 09:45:00.515836 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:00.515808    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-544f748fdb-5pr6x"
Sep 14 09:45:00.534902 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:00.534867    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2ed75443-bd9e-40ad-8a4f-b9047cce9f53-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-79c94f89b6-6stck\" (UID: \"2ed75443-bd9e-40ad-8a4f-b9047cce9f53\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79c94f89b6-6stck"
Sep 14 09:45:00.535140 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:00.534923    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7zdq\" (UniqueName: \"kubernetes.io/projected/2ed75443-bd9e-40ad-8a4f-b9047cce9f53-kube-api-access-f7zdq\") pod \"managed-serviceaccount-addon-agent-79c94f89b6-6stck\" (UID: \"2ed75443-bd9e-40ad-8a4f-b9047cce9f53\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79c94f89b6-6stck"
Sep 14 09:45:00.537632 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:00.537598    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2ed75443-bd9e-40ad-8a4f-b9047cce9f53-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-79c94f89b6-6stck\" (UID: \"2ed75443-bd9e-40ad-8a4f-b9047cce9f53\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79c94f89b6-6stck"
Sep 14 09:45:00.571183 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:00.571146    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-544f748fdb-5pr6x"]
Sep 14 09:45:00.581622 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:00.581589    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7zdq\" (UniqueName: \"kubernetes.io/projected/2ed75443-bd9e-40ad-8a4f-b9047cce9f53-kube-api-access-f7zdq\") pod \"managed-serviceaccount-addon-agent-79c94f89b6-6stck\" (UID: \"2ed75443-bd9e-40ad-8a4f-b9047cce9f53\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79c94f89b6-6stck"
Sep 14 09:45:00.635723 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:00.635618    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f610700e-43a1-443a-a589-3420e9ce4c91-registry-certificates\") pod \"image-registry-544f748fdb-5pr6x\" (UID: \"f610700e-43a1-443a-a589-3420e9ce4c91\") " pod="openshift-image-registry/image-registry-544f748fdb-5pr6x"
Sep 14 09:45:00.635723 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:00.635674    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f610700e-43a1-443a-a589-3420e9ce4c91-image-registry-private-configuration\") pod \"image-registry-544f748fdb-5pr6x\" (UID: \"f610700e-43a1-443a-a589-3420e9ce4c91\") " pod="openshift-image-registry/image-registry-544f748fdb-5pr6x"
Sep 14 09:45:00.635723 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:00.635703    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f610700e-43a1-443a-a589-3420e9ce4c91-trusted-ca\") pod \"image-registry-544f748fdb-5pr6x\" (UID: \"f610700e-43a1-443a-a589-3420e9ce4c91\") " pod="openshift-image-registry/image-registry-544f748fdb-5pr6x"
Sep 14 09:45:00.635962 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:00.635754    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f610700e-43a1-443a-a589-3420e9ce4c91-installation-pull-secrets\") pod \"image-registry-544f748fdb-5pr6x\" (UID: \"f610700e-43a1-443a-a589-3420e9ce4c91\") " pod="openshift-image-registry/image-registry-544f748fdb-5pr6x"
Sep 14 09:45:00.635962 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:00.635774    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f610700e-43a1-443a-a589-3420e9ce4c91-registry-tls\") pod \"image-registry-544f748fdb-5pr6x\" (UID: \"f610700e-43a1-443a-a589-3420e9ce4c91\") " pod="openshift-image-registry/image-registry-544f748fdb-5pr6x"
Sep 14 09:45:00.635962 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:00.635808    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpwmm\" (UniqueName: \"kubernetes.io/projected/f610700e-43a1-443a-a589-3420e9ce4c91-kube-api-access-wpwmm\") pod \"image-registry-544f748fdb-5pr6x\" (UID: \"f610700e-43a1-443a-a589-3420e9ce4c91\") " pod="openshift-image-registry/image-registry-544f748fdb-5pr6x"
Sep 14 09:45:00.635962 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:00.635826    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f610700e-43a1-443a-a589-3420e9ce4c91-ca-trust-extracted\") pod \"image-registry-544f748fdb-5pr6x\" (UID: \"f610700e-43a1-443a-a589-3420e9ce4c91\") " pod="openshift-image-registry/image-registry-544f748fdb-5pr6x"
Sep 14 09:45:00.635962 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:00.635929    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f610700e-43a1-443a-a589-3420e9ce4c91-bound-sa-token\") pod \"image-registry-544f748fdb-5pr6x\" (UID: \"f610700e-43a1-443a-a589-3420e9ce4c91\") " pod="openshift-image-registry/image-registry-544f748fdb-5pr6x"
Sep 14 09:45:00.673197 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:00.673165    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:00.697059 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:00.697025    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79c94f89b6-6stck"
Sep 14 09:45:00.737313 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:00.737269    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f610700e-43a1-443a-a589-3420e9ce4c91-bound-sa-token\") pod \"image-registry-544f748fdb-5pr6x\" (UID: \"f610700e-43a1-443a-a589-3420e9ce4c91\") " pod="openshift-image-registry/image-registry-544f748fdb-5pr6x"
Sep 14 09:45:00.737313 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:00.737313    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f610700e-43a1-443a-a589-3420e9ce4c91-registry-certificates\") pod \"image-registry-544f748fdb-5pr6x\" (UID: \"f610700e-43a1-443a-a589-3420e9ce4c91\") " pod="openshift-image-registry/image-registry-544f748fdb-5pr6x"
Sep 14 09:45:00.737568 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:00.737457    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f610700e-43a1-443a-a589-3420e9ce4c91-image-registry-private-configuration\") pod \"image-registry-544f748fdb-5pr6x\" (UID: \"f610700e-43a1-443a-a589-3420e9ce4c91\") " pod="openshift-image-registry/image-registry-544f748fdb-5pr6x"
Sep 14 09:45:00.737568 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:00.737505    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f610700e-43a1-443a-a589-3420e9ce4c91-trusted-ca\") pod \"image-registry-544f748fdb-5pr6x\" (UID: \"f610700e-43a1-443a-a589-3420e9ce4c91\") " pod="openshift-image-registry/image-registry-544f748fdb-5pr6x"
Sep 14 09:45:00.737568 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:00.737558    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f610700e-43a1-443a-a589-3420e9ce4c91-installation-pull-secrets\") pod \"image-registry-544f748fdb-5pr6x\" (UID: \"f610700e-43a1-443a-a589-3420e9ce4c91\") " pod="openshift-image-registry/image-registry-544f748fdb-5pr6x"
Sep 14 09:45:00.737709 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:00.737592    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f610700e-43a1-443a-a589-3420e9ce4c91-registry-tls\") pod \"image-registry-544f748fdb-5pr6x\" (UID: \"f610700e-43a1-443a-a589-3420e9ce4c91\") " pod="openshift-image-registry/image-registry-544f748fdb-5pr6x"
Sep 14 09:45:00.737709 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:00.737665    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpwmm\" (UniqueName: \"kubernetes.io/projected/f610700e-43a1-443a-a589-3420e9ce4c91-kube-api-access-wpwmm\") pod \"image-registry-544f748fdb-5pr6x\" (UID: \"f610700e-43a1-443a-a589-3420e9ce4c91\") " pod="openshift-image-registry/image-registry-544f748fdb-5pr6x"
Sep 14 09:45:00.737709 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:00.737697    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f610700e-43a1-443a-a589-3420e9ce4c91-ca-trust-extracted\") pod \"image-registry-544f748fdb-5pr6x\" (UID: \"f610700e-43a1-443a-a589-3420e9ce4c91\") " pod="openshift-image-registry/image-registry-544f748fdb-5pr6x"
Sep 14 09:45:00.738398 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:00.738062    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f610700e-43a1-443a-a589-3420e9ce4c91-ca-trust-extracted\") pod \"image-registry-544f748fdb-5pr6x\" (UID: \"f610700e-43a1-443a-a589-3420e9ce4c91\") " pod="openshift-image-registry/image-registry-544f748fdb-5pr6x"
Sep 14 09:45:00.738398 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:00.738116    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f610700e-43a1-443a-a589-3420e9ce4c91-registry-certificates\") pod \"image-registry-544f748fdb-5pr6x\" (UID: \"f610700e-43a1-443a-a589-3420e9ce4c91\") " pod="openshift-image-registry/image-registry-544f748fdb-5pr6x"
Sep 14 09:45:00.738802 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:00.738776    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f610700e-43a1-443a-a589-3420e9ce4c91-trusted-ca\") pod \"image-registry-544f748fdb-5pr6x\" (UID: \"f610700e-43a1-443a-a589-3420e9ce4c91\") " pod="openshift-image-registry/image-registry-544f748fdb-5pr6x"
Sep 14 09:45:00.740533 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:00.740510    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f610700e-43a1-443a-a589-3420e9ce4c91-image-registry-private-configuration\") pod \"image-registry-544f748fdb-5pr6x\" (UID: \"f610700e-43a1-443a-a589-3420e9ce4c91\") " pod="openshift-image-registry/image-registry-544f748fdb-5pr6x"
Sep 14 09:45:00.740641 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:00.740615    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f610700e-43a1-443a-a589-3420e9ce4c91-installation-pull-secrets\") pod \"image-registry-544f748fdb-5pr6x\" (UID: \"f610700e-43a1-443a-a589-3420e9ce4c91\") " pod="openshift-image-registry/image-registry-544f748fdb-5pr6x"
Sep 14 09:45:00.740716 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:00.740697    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f610700e-43a1-443a-a589-3420e9ce4c91-registry-tls\") pod \"image-registry-544f748fdb-5pr6x\" (UID: \"f610700e-43a1-443a-a589-3420e9ce4c91\") " pod="openshift-image-registry/image-registry-544f748fdb-5pr6x"
Sep 14 09:45:00.747271 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:00.747231    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpwmm\" (UniqueName: \"kubernetes.io/projected/f610700e-43a1-443a-a589-3420e9ce4c91-kube-api-access-wpwmm\") pod \"image-registry-544f748fdb-5pr6x\" (UID: \"f610700e-43a1-443a-a589-3420e9ce4c91\") " pod="openshift-image-registry/image-registry-544f748fdb-5pr6x"
Sep 14 09:45:00.749377 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:00.749325    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f610700e-43a1-443a-a589-3420e9ce4c91-bound-sa-token\") pod \"image-registry-544f748fdb-5pr6x\" (UID: \"f610700e-43a1-443a-a589-3420e9ce4c91\") " pod="openshift-image-registry/image-registry-544f748fdb-5pr6x"
Sep 14 09:45:00.775041 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:00.774691    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:00.825270 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:00.825228    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-544f748fdb-5pr6x"
Sep 14 09:45:01.001395 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:01.001362    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79c94f89b6-6stck"]
Sep 14 09:45:01.004824 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:45:01.004791    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2ed75443_bd9e_40ad_8a4f_b9047cce9f53.slice/crio-04f62894e3ee5276b96a458a84cde852fcded037aee99549270822044c7181c7 WatchSource:0}: Error finding container 04f62894e3ee5276b96a458a84cde852fcded037aee99549270822044c7181c7: Status 404 returned error can't find the container with id 04f62894e3ee5276b96a458a84cde852fcded037aee99549270822044c7181c7
Sep 14 09:45:01.110204 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:01.110180    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-544f748fdb-5pr6x"]
Sep 14 09:45:01.112773 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:45:01.112749    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf610700e_43a1_443a_a589_3420e9ce4c91.slice/crio-9ed115c104bfd48ae22d93989786b8029b15d386963ee21a92670d1b2fa8fd90 WatchSource:0}: Error finding container 9ed115c104bfd48ae22d93989786b8029b15d386963ee21a92670d1b2fa8fd90: Status 404 returned error can't find the container with id 9ed115c104bfd48ae22d93989786b8029b15d386963ee21a92670d1b2fa8fd90
Sep 14 09:45:01.589151 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:01.589111    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:01.691312 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:01.691257    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:01.984902 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:01.984864    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-544f748fdb-5pr6x" event={"ID":"f610700e-43a1-443a-a589-3420e9ce4c91","Type":"ContainerStarted","Data":"d9f6a271774b8098069dd493b14375c0ede9019112e05dd35f3c84bdde0ac509"}
Sep 14 09:45:01.985127 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:01.984935    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-544f748fdb-5pr6x"
Sep 14 09:45:01.985127 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:01.984958    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-544f748fdb-5pr6x" event={"ID":"f610700e-43a1-443a-a589-3420e9ce4c91","Type":"ContainerStarted","Data":"9ed115c104bfd48ae22d93989786b8029b15d386963ee21a92670d1b2fa8fd90"}
Sep 14 09:45:01.986482 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:01.986376    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79c94f89b6-6stck" event={"ID":"2ed75443-bd9e-40ad-8a4f-b9047cce9f53","Type":"ContainerStarted","Data":"04f62894e3ee5276b96a458a84cde852fcded037aee99549270822044c7181c7"}
Sep 14 09:45:02.008440 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:02.008359    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-544f748fdb-5pr6x" podStartSLOduration=2.008332095 podStartE2EDuration="2.008332095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:45:02.007469773 +0000 UTC m=+91.047144395" watchObservedRunningTime="2026-09-14 09:45:02.008332095 +0000 UTC m=+91.048006718"
Sep 14 09:45:02.757306 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:02.757252    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/161e5bed-e6f8-4c08-8c69-857064cc8c4f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-7tpzw\" (UID: \"161e5bed-e6f8-4c08-8c69-857064cc8c4f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-7tpzw"
Sep 14 09:45:02.760227 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:02.760193    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/161e5bed-e6f8-4c08-8c69-857064cc8c4f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-7tpzw\" (UID: \"161e5bed-e6f8-4c08-8c69-857064cc8c4f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-7tpzw"
Sep 14 09:45:02.995140 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:02.993971    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-nvtsx\""
Sep 14 09:45:02.995775 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:02.995733    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-7tpzw"
Sep 14 09:45:03.907745 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:03.907654    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-7tpzw"]
Sep 14 09:45:03.911125 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:45:03.911094    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod161e5bed_e6f8_4c08_8c69_857064cc8c4f.slice/crio-8b77dafa00439cb6ae9aa7c4fbd5451bb2dab04a4d8aeb8493cb9ad353bd401e WatchSource:0}: Error finding container 8b77dafa00439cb6ae9aa7c4fbd5451bb2dab04a4d8aeb8493cb9ad353bd401e: Status 404 returned error can't find the container with id 8b77dafa00439cb6ae9aa7c4fbd5451bb2dab04a4d8aeb8493cb9ad353bd401e
Sep 14 09:45:04.000981 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:04.000940    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-7tpzw" event={"ID":"161e5bed-e6f8-4c08-8c69-857064cc8c4f","Type":"ContainerStarted","Data":"8b77dafa00439cb6ae9aa7c4fbd5451bb2dab04a4d8aeb8493cb9ad353bd401e"}
Sep 14 09:45:04.002449 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:04.002413    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79c94f89b6-6stck" event={"ID":"2ed75443-bd9e-40ad-8a4f-b9047cce9f53","Type":"ContainerStarted","Data":"ef00a85eff8843e42aedac513666a3636d7e7c059a0ee68cd8e5a2b950b9048a"}
Sep 14 09:45:04.018789 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:04.018737    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79c94f89b6-6stck" podStartSLOduration=1.325715727 podStartE2EDuration="4.0187196s" totalImagesPullingTime="2.693003873s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:45:04.018530132 +0000 UTC m=+93.058204754" watchObservedRunningTime="2026-09-14 09:45:04.0187196 +0000 UTC m=+93.058394231"
Sep 14 09:45:05.519912 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:05.519857    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:05.613184 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:05.613147    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:05.856899 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:05.856772    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:05.951864 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:05.951830    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:06.012380 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:06.012330    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-7tpzw" event={"ID":"161e5bed-e6f8-4c08-8c69-857064cc8c4f","Type":"ContainerStarted","Data":"aa54c175d131f652bc5f27263fcaaa61971cde17e7c46f29da15e6a330a9a7b1"}
Sep 14 09:45:06.039913 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:06.039853    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-7tpzw" podStartSLOduration=34.26215981 podStartE2EDuration="36.039833678s" totalImagesPullingTime="1.777673868s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:44:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:45:06.038771382 +0000 UTC m=+95.078446005" watchObservedRunningTime="2026-09-14 09:45:06.039833678 +0000 UTC m=+95.079508299"
Sep 14 09:45:06.081753 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:06.081714    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:06.160823 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:06.160728    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:06.244548 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:06.244502    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:06.492175 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:06.492127    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:07.295739 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:07.295703    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:07.361666 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:07.361628    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:07.546378 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:07.546290    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-74f9bbb8b6-nlfpv"]
Sep 14 09:45:07.551976 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:07.551943    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74f9bbb8b6-nlfpv"
Sep 14 09:45:07.552460 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:07.552413    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:07.554828 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:07.554807    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 14 09:45:07.554939 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:07.554902    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 14 09:45:07.555496 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:07.555479    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 14 09:45:07.555812 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:07.555793    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 14 09:45:07.556109 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:07.556085    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 14 09:45:07.556330 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:07.556187    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 14 09:45:07.556410 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:07.556219    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-stffz\""
Sep 14 09:45:07.556470 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:07.556219    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 14 09:45:07.560693 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:07.560664    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74f9bbb8b6-nlfpv"]
Sep 14 09:45:07.562273 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:07.562248    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 14 09:45:07.698270 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:07.698223    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f9e7a90-aded-4d16-9af6-bde1bccc0002-console-oauth-config\") pod \"console-74f9bbb8b6-nlfpv\" (UID: \"8f9e7a90-aded-4d16-9af6-bde1bccc0002\") " pod="openshift-console/console-74f9bbb8b6-nlfpv"
Sep 14 09:45:07.698270 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:07.698263    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f9e7a90-aded-4d16-9af6-bde1bccc0002-trusted-ca-bundle\") pod \"console-74f9bbb8b6-nlfpv\" (UID: \"8f9e7a90-aded-4d16-9af6-bde1bccc0002\") " pod="openshift-console/console-74f9bbb8b6-nlfpv"
Sep 14 09:45:07.698500 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:07.698286    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f9e7a90-aded-4d16-9af6-bde1bccc0002-service-ca\") pod \"console-74f9bbb8b6-nlfpv\" (UID: \"8f9e7a90-aded-4d16-9af6-bde1bccc0002\") " pod="openshift-console/console-74f9bbb8b6-nlfpv"
Sep 14 09:45:07.698500 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:07.698376    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f9e7a90-aded-4d16-9af6-bde1bccc0002-console-serving-cert\") pod \"console-74f9bbb8b6-nlfpv\" (UID: \"8f9e7a90-aded-4d16-9af6-bde1bccc0002\") " pod="openshift-console/console-74f9bbb8b6-nlfpv"
Sep 14 09:45:07.698500 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:07.698410    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44mq2\" (UniqueName: \"kubernetes.io/projected/8f9e7a90-aded-4d16-9af6-bde1bccc0002-kube-api-access-44mq2\") pod \"console-74f9bbb8b6-nlfpv\" (UID: \"8f9e7a90-aded-4d16-9af6-bde1bccc0002\") " pod="openshift-console/console-74f9bbb8b6-nlfpv"
Sep 14 09:45:07.698500 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:07.698426    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8f9e7a90-aded-4d16-9af6-bde1bccc0002-console-config\") pod \"console-74f9bbb8b6-nlfpv\" (UID: \"8f9e7a90-aded-4d16-9af6-bde1bccc0002\") " pod="openshift-console/console-74f9bbb8b6-nlfpv"
Sep 14 09:45:07.698500 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:07.698459    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f9e7a90-aded-4d16-9af6-bde1bccc0002-oauth-serving-cert\") pod \"console-74f9bbb8b6-nlfpv\" (UID: \"8f9e7a90-aded-4d16-9af6-bde1bccc0002\") " pod="openshift-console/console-74f9bbb8b6-nlfpv"
Sep 14 09:45:07.798930 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:07.798828    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f9e7a90-aded-4d16-9af6-bde1bccc0002-console-oauth-config\") pod \"console-74f9bbb8b6-nlfpv\" (UID: \"8f9e7a90-aded-4d16-9af6-bde1bccc0002\") " pod="openshift-console/console-74f9bbb8b6-nlfpv"
Sep 14 09:45:07.798930 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:07.798870    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f9e7a90-aded-4d16-9af6-bde1bccc0002-trusted-ca-bundle\") pod \"console-74f9bbb8b6-nlfpv\" (UID: \"8f9e7a90-aded-4d16-9af6-bde1bccc0002\") " pod="openshift-console/console-74f9bbb8b6-nlfpv"
Sep 14 09:45:07.798930 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:07.798898    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f9e7a90-aded-4d16-9af6-bde1bccc0002-service-ca\") pod \"console-74f9bbb8b6-nlfpv\" (UID: \"8f9e7a90-aded-4d16-9af6-bde1bccc0002\") " pod="openshift-console/console-74f9bbb8b6-nlfpv"
Sep 14 09:45:07.799239 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:07.798941    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f9e7a90-aded-4d16-9af6-bde1bccc0002-console-serving-cert\") pod \"console-74f9bbb8b6-nlfpv\" (UID: \"8f9e7a90-aded-4d16-9af6-bde1bccc0002\") " pod="openshift-console/console-74f9bbb8b6-nlfpv"
Sep 14 09:45:07.799239 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:07.798966    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44mq2\" (UniqueName: \"kubernetes.io/projected/8f9e7a90-aded-4d16-9af6-bde1bccc0002-kube-api-access-44mq2\") pod \"console-74f9bbb8b6-nlfpv\" (UID: \"8f9e7a90-aded-4d16-9af6-bde1bccc0002\") " pod="openshift-console/console-74f9bbb8b6-nlfpv"
Sep 14 09:45:07.799239 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:07.798989    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8f9e7a90-aded-4d16-9af6-bde1bccc0002-console-config\") pod \"console-74f9bbb8b6-nlfpv\" (UID: \"8f9e7a90-aded-4d16-9af6-bde1bccc0002\") " pod="openshift-console/console-74f9bbb8b6-nlfpv"
Sep 14 09:45:07.799239 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:07.799053    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f9e7a90-aded-4d16-9af6-bde1bccc0002-oauth-serving-cert\") pod \"console-74f9bbb8b6-nlfpv\" (UID: \"8f9e7a90-aded-4d16-9af6-bde1bccc0002\") " pod="openshift-console/console-74f9bbb8b6-nlfpv"
Sep 14 09:45:07.799803 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:07.799774    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f9e7a90-aded-4d16-9af6-bde1bccc0002-service-ca\") pod \"console-74f9bbb8b6-nlfpv\" (UID: \"8f9e7a90-aded-4d16-9af6-bde1bccc0002\") " pod="openshift-console/console-74f9bbb8b6-nlfpv"
Sep 14 09:45:07.799921 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:07.799865    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f9e7a90-aded-4d16-9af6-bde1bccc0002-oauth-serving-cert\") pod \"console-74f9bbb8b6-nlfpv\" (UID: \"8f9e7a90-aded-4d16-9af6-bde1bccc0002\") " pod="openshift-console/console-74f9bbb8b6-nlfpv"
Sep 14 09:45:07.799981 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:07.799921    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8f9e7a90-aded-4d16-9af6-bde1bccc0002-console-config\") pod \"console-74f9bbb8b6-nlfpv\" (UID: \"8f9e7a90-aded-4d16-9af6-bde1bccc0002\") " pod="openshift-console/console-74f9bbb8b6-nlfpv"
Sep 14 09:45:07.800081 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:07.800061    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f9e7a90-aded-4d16-9af6-bde1bccc0002-trusted-ca-bundle\") pod \"console-74f9bbb8b6-nlfpv\" (UID: \"8f9e7a90-aded-4d16-9af6-bde1bccc0002\") " pod="openshift-console/console-74f9bbb8b6-nlfpv"
Sep 14 09:45:07.801527 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:07.801504    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f9e7a90-aded-4d16-9af6-bde1bccc0002-console-serving-cert\") pod \"console-74f9bbb8b6-nlfpv\" (UID: \"8f9e7a90-aded-4d16-9af6-bde1bccc0002\") " pod="openshift-console/console-74f9bbb8b6-nlfpv"
Sep 14 09:45:07.801636 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:07.801508    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f9e7a90-aded-4d16-9af6-bde1bccc0002-console-oauth-config\") pod \"console-74f9bbb8b6-nlfpv\" (UID: \"8f9e7a90-aded-4d16-9af6-bde1bccc0002\") " pod="openshift-console/console-74f9bbb8b6-nlfpv"
Sep 14 09:45:07.808040 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:07.807988    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44mq2\" (UniqueName: \"kubernetes.io/projected/8f9e7a90-aded-4d16-9af6-bde1bccc0002-kube-api-access-44mq2\") pod \"console-74f9bbb8b6-nlfpv\" (UID: \"8f9e7a90-aded-4d16-9af6-bde1bccc0002\") " pod="openshift-console/console-74f9bbb8b6-nlfpv"
Sep 14 09:45:07.862303 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:07.862264    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74f9bbb8b6-nlfpv"
Sep 14 09:45:08.138313 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:08.138262    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74f9bbb8b6-nlfpv"]
Sep 14 09:45:08.140736 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:45:08.140705    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f9e7a90_aded_4d16_9af6_bde1bccc0002.slice/crio-9a7b7dfb8bcfdc7841032d2702b2c986ce8522fe3969913a7d82731c22f461e3 WatchSource:0}: Error finding container 9a7b7dfb8bcfdc7841032d2702b2c986ce8522fe3969913a7d82731c22f461e3: Status 404 returned error can't find the container with id 9a7b7dfb8bcfdc7841032d2702b2c986ce8522fe3969913a7d82731c22f461e3
Sep 14 09:45:09.024675 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:09.024636    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74f9bbb8b6-nlfpv" event={"ID":"8f9e7a90-aded-4d16-9af6-bde1bccc0002","Type":"ContainerStarted","Data":"9a7b7dfb8bcfdc7841032d2702b2c986ce8522fe3969913a7d82731c22f461e3"}
Sep 14 09:45:09.715343 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:09.715298    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec1015e1-7070-4f02-8f4e-2fd5d395c4e9-metrics-tls\") pod \"dns-default-mklrw\" (UID: \"ec1015e1-7070-4f02-8f4e-2fd5d395c4e9\") " pod="openshift-dns/dns-default-mklrw"
Sep 14 09:45:09.718124 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:09.718090    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec1015e1-7070-4f02-8f4e-2fd5d395c4e9-metrics-tls\") pod \"dns-default-mklrw\" (UID: \"ec1015e1-7070-4f02-8f4e-2fd5d395c4e9\") " pod="openshift-dns/dns-default-mklrw"
Sep 14 09:45:09.958231 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:09.958193    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hwxbr\""
Sep 14 09:45:09.966392 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:09.966297    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mklrw"
Sep 14 09:45:10.419701 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:10.419666    2563 patch_prober.go:28] interesting pod/image-registry-9689ffdd5-8254l container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 14 09:45:10.420112 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:10.419727    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-9689ffdd5-8254l" podUID="cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 14 09:45:10.884719 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:10.884684    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mklrw"]
Sep 14 09:45:10.888032 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:45:10.887988    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec1015e1_7070_4f02_8f4e_2fd5d395c4e9.slice/crio-5a62c44c38da06dce6a1f262e63928d19e3a914a4c5402f7f7286e5bfa124cc3 WatchSource:0}: Error finding container 5a62c44c38da06dce6a1f262e63928d19e3a914a4c5402f7f7286e5bfa124cc3: Status 404 returned error can't find the container with id 5a62c44c38da06dce6a1f262e63928d19e3a914a4c5402f7f7286e5bfa124cc3
Sep 14 09:45:11.033849 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:11.033811    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74f9bbb8b6-nlfpv" event={"ID":"8f9e7a90-aded-4d16-9af6-bde1bccc0002","Type":"ContainerStarted","Data":"00e232da119e791f26180bcd91ca511edc526cb8d840994dbd1cc3e6c90b1f90"}
Sep 14 09:45:11.035066 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:11.035040    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mklrw" event={"ID":"ec1015e1-7070-4f02-8f4e-2fd5d395c4e9","Type":"ContainerStarted","Data":"5a62c44c38da06dce6a1f262e63928d19e3a914a4c5402f7f7286e5bfa124cc3"}
Sep 14 09:45:11.069314 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:11.069261    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-74f9bbb8b6-nlfpv" podStartSLOduration=1.546251773 podStartE2EDuration="4.069244172s" totalImagesPullingTime="2.522992399s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:45:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:45:11.068501564 +0000 UTC m=+100.108176186" watchObservedRunningTime="2026-09-14 09:45:11.069244172 +0000 UTC m=+100.108918847"
Sep 14 09:45:11.259394 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:11.259361    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:11.339830 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:11.339736    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:11.430664 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:11.430615    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:11.540065 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:11.540025    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:13.044438 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:13.044398    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mklrw" event={"ID":"ec1015e1-7070-4f02-8f4e-2fd5d395c4e9","Type":"ContainerStarted","Data":"600a5a4cdadf6b8fe099a76b717ac9cf60dde11c3eea64d9f77188de1bbe91cd"}
Sep 14 09:45:13.044818 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:13.044465    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mklrw" event={"ID":"ec1015e1-7070-4f02-8f4e-2fd5d395c4e9","Type":"ContainerStarted","Data":"1fdd58982cf5e09f7929216f6d65a3d8ed6a629e2746fc3b8c6858ceb8a8b75b"}
Sep 14 09:45:13.044818 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:13.044548    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-mklrw"
Sep 14 09:45:13.063521 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:13.063461    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-mklrw" podStartSLOduration=66.363046212 podStartE2EDuration="1m8.063440337s" totalImagesPullingTime="1.700394125s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:44:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:45:13.061943336 +0000 UTC m=+102.101617958" watchObservedRunningTime="2026-09-14 09:45:13.063440337 +0000 UTC m=+102.103114958"
Sep 14 09:45:13.907680 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:13.907636    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8zcts"
Sep 14 09:45:14.075452 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:14.075411    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:15.346345 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.346305    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:15.449669 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.449630    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:15.547414 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.547374    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:15.647783 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.647684    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:15.669016 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.668972    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-ldtq5"]
Sep 14 09:45:15.703864 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.703820    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-ldtq5"]
Sep 14 09:45:15.703864 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.703874    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-v8jjn"]
Sep 14 09:45:15.704097 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.703959    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-ldtq5"
Sep 14 09:45:15.706192 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.706165    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-sdmlp\""
Sep 14 09:45:15.706869 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.706836    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 14 09:45:15.707026 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.706992    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 14 09:45:15.707111 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.707034    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 14 09:45:15.717178 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.717144    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-zkffp"]
Sep 14 09:45:15.717387 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.717366    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v8jjn"
Sep 14 09:45:15.719525 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.719492    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-d9gll\""
Sep 14 09:45:15.719815 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.719795    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 14 09:45:15.719969 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.719857    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 14 09:45:15.719969 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.719902    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 14 09:45:15.739222 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.739189    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-v8jjn"]
Sep 14 09:45:15.739381 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.739329    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zkffp"
Sep 14 09:45:15.741609 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.741583    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 14 09:45:15.741785 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.741747    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 14 09:45:15.741881 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.741865    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ghd85\""
Sep 14 09:45:15.741959 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.741872    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 14 09:45:15.749048 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.749021    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:15.848552 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.848522    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:15.867230 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.867191    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqdbm\" (UniqueName: \"kubernetes.io/projected/698195e3-0a13-4176-ab4c-f8167e92eefb-kube-api-access-tqdbm\") pod \"openshift-state-metrics-67dcf8c979-ldtq5\" (UID: \"698195e3-0a13-4176-ab4c-f8167e92eefb\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-ldtq5"
Sep 14 09:45:15.867398 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.867236    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/698195e3-0a13-4176-ab4c-f8167e92eefb-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-ldtq5\" (UID: \"698195e3-0a13-4176-ab4c-f8167e92eefb\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-ldtq5"
Sep 14 09:45:15.867398 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.867272    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dabf7719-e039-48a1-8150-f100f09e2d4a-node-exporter-textfile\") pod \"node-exporter-zkffp\" (UID: \"dabf7719-e039-48a1-8150-f100f09e2d4a\") " pod="openshift-monitoring/node-exporter-zkffp"
Sep 14 09:45:15.867398 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.867298    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/935ba669-ed70-406a-a645-054a8db8b24b-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-v8jjn\" (UID: \"935ba669-ed70-406a-a645-054a8db8b24b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v8jjn"
Sep 14 09:45:15.867398 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.867326    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/698195e3-0a13-4176-ab4c-f8167e92eefb-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-ldtq5\" (UID: \"698195e3-0a13-4176-ab4c-f8167e92eefb\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-ldtq5"
Sep 14 09:45:15.867625 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.867400    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/935ba669-ed70-406a-a645-054a8db8b24b-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-v8jjn\" (UID: \"935ba669-ed70-406a-a645-054a8db8b24b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v8jjn"
Sep 14 09:45:15.867625 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.867447    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dabf7719-e039-48a1-8150-f100f09e2d4a-root\") pod \"node-exporter-zkffp\" (UID: \"dabf7719-e039-48a1-8150-f100f09e2d4a\") " pod="openshift-monitoring/node-exporter-zkffp"
Sep 14 09:45:15.867625 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.867504    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dabf7719-e039-48a1-8150-f100f09e2d4a-sys\") pod \"node-exporter-zkffp\" (UID: \"dabf7719-e039-48a1-8150-f100f09e2d4a\") " pod="openshift-monitoring/node-exporter-zkffp"
Sep 14 09:45:15.867625 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.867533    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/935ba669-ed70-406a-a645-054a8db8b24b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-v8jjn\" (UID: \"935ba669-ed70-406a-a645-054a8db8b24b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v8jjn"
Sep 14 09:45:15.867625 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.867573    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rppr\" (UniqueName: \"kubernetes.io/projected/dabf7719-e039-48a1-8150-f100f09e2d4a-kube-api-access-7rppr\") pod \"node-exporter-zkffp\" (UID: \"dabf7719-e039-48a1-8150-f100f09e2d4a\") " pod="openshift-monitoring/node-exporter-zkffp"
Sep 14 09:45:15.867625 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.867612    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/935ba669-ed70-406a-a645-054a8db8b24b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-v8jjn\" (UID: \"935ba669-ed70-406a-a645-054a8db8b24b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v8jjn"
Sep 14 09:45:15.867829 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.867633    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dabf7719-e039-48a1-8150-f100f09e2d4a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zkffp\" (UID: \"dabf7719-e039-48a1-8150-f100f09e2d4a\") " pod="openshift-monitoring/node-exporter-zkffp"
Sep 14 09:45:15.867829 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.867666    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dabf7719-e039-48a1-8150-f100f09e2d4a-node-exporter-wtmp\") pod \"node-exporter-zkffp\" (UID: \"dabf7719-e039-48a1-8150-f100f09e2d4a\") " pod="openshift-monitoring/node-exporter-zkffp"
Sep 14 09:45:15.867829 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.867688    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mk8c\" (UniqueName: \"kubernetes.io/projected/935ba669-ed70-406a-a645-054a8db8b24b-kube-api-access-9mk8c\") pod \"kube-state-metrics-59fbdd77d-v8jjn\" (UID: \"935ba669-ed70-406a-a645-054a8db8b24b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v8jjn"
Sep 14 09:45:15.867829 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.867703    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dabf7719-e039-48a1-8150-f100f09e2d4a-node-exporter-tls\") pod \"node-exporter-zkffp\" (UID: \"dabf7719-e039-48a1-8150-f100f09e2d4a\") " pod="openshift-monitoring/node-exporter-zkffp"
Sep 14 09:45:15.867829 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.867720    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dabf7719-e039-48a1-8150-f100f09e2d4a-node-exporter-accelerators-collector-config\") pod \"node-exporter-zkffp\" (UID: \"dabf7719-e039-48a1-8150-f100f09e2d4a\") " pod="openshift-monitoring/node-exporter-zkffp"
Sep 14 09:45:15.867829 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.867746    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/698195e3-0a13-4176-ab4c-f8167e92eefb-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-ldtq5\" (UID: \"698195e3-0a13-4176-ab4c-f8167e92eefb\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-ldtq5"
Sep 14 09:45:15.867829 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.867770    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/935ba669-ed70-406a-a645-054a8db8b24b-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-v8jjn\" (UID: \"935ba669-ed70-406a-a645-054a8db8b24b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v8jjn"
Sep 14 09:45:15.867829 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.867812    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dabf7719-e039-48a1-8150-f100f09e2d4a-metrics-client-ca\") pod \"node-exporter-zkffp\" (UID: \"dabf7719-e039-48a1-8150-f100f09e2d4a\") " pod="openshift-monitoring/node-exporter-zkffp"
Sep 14 09:45:15.949790 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.949696    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:15.968445 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.968401    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/935ba669-ed70-406a-a645-054a8db8b24b-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-v8jjn\" (UID: \"935ba669-ed70-406a-a645-054a8db8b24b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v8jjn"
Sep 14 09:45:15.968445 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.968444    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dabf7719-e039-48a1-8150-f100f09e2d4a-root\") pod \"node-exporter-zkffp\" (UID: \"dabf7719-e039-48a1-8150-f100f09e2d4a\") " pod="openshift-monitoring/node-exporter-zkffp"
Sep 14 09:45:15.968684 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.968484    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dabf7719-e039-48a1-8150-f100f09e2d4a-sys\") pod \"node-exporter-zkffp\" (UID: \"dabf7719-e039-48a1-8150-f100f09e2d4a\") " pod="openshift-monitoring/node-exporter-zkffp"
Sep 14 09:45:15.968684 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.968501    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/935ba669-ed70-406a-a645-054a8db8b24b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-v8jjn\" (UID: \"935ba669-ed70-406a-a645-054a8db8b24b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v8jjn"
Sep 14 09:45:15.968684 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.968526    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rppr\" (UniqueName: \"kubernetes.io/projected/dabf7719-e039-48a1-8150-f100f09e2d4a-kube-api-access-7rppr\") pod \"node-exporter-zkffp\" (UID: \"dabf7719-e039-48a1-8150-f100f09e2d4a\") " pod="openshift-monitoring/node-exporter-zkffp"
Sep 14 09:45:15.968684 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.968557    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/935ba669-ed70-406a-a645-054a8db8b24b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-v8jjn\" (UID: \"935ba669-ed70-406a-a645-054a8db8b24b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v8jjn"
Sep 14 09:45:15.968684 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.968597    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dabf7719-e039-48a1-8150-f100f09e2d4a-sys\") pod \"node-exporter-zkffp\" (UID: \"dabf7719-e039-48a1-8150-f100f09e2d4a\") " pod="openshift-monitoring/node-exporter-zkffp"
Sep 14 09:45:15.968684 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.968620    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dabf7719-e039-48a1-8150-f100f09e2d4a-root\") pod \"node-exporter-zkffp\" (UID: \"dabf7719-e039-48a1-8150-f100f09e2d4a\") " pod="openshift-monitoring/node-exporter-zkffp"
Sep 14 09:45:15.968684 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.968679    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dabf7719-e039-48a1-8150-f100f09e2d4a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zkffp\" (UID: \"dabf7719-e039-48a1-8150-f100f09e2d4a\") " pod="openshift-monitoring/node-exporter-zkffp"
Sep 14 09:45:15.969049 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.968722    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dabf7719-e039-48a1-8150-f100f09e2d4a-node-exporter-wtmp\") pod \"node-exporter-zkffp\" (UID: \"dabf7719-e039-48a1-8150-f100f09e2d4a\") " pod="openshift-monitoring/node-exporter-zkffp"
Sep 14 09:45:15.969049 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.968747    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mk8c\" (UniqueName: \"kubernetes.io/projected/935ba669-ed70-406a-a645-054a8db8b24b-kube-api-access-9mk8c\") pod \"kube-state-metrics-59fbdd77d-v8jjn\" (UID: \"935ba669-ed70-406a-a645-054a8db8b24b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v8jjn"
Sep 14 09:45:15.969049 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.968763    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dabf7719-e039-48a1-8150-f100f09e2d4a-node-exporter-tls\") pod \"node-exporter-zkffp\" (UID: \"dabf7719-e039-48a1-8150-f100f09e2d4a\") " pod="openshift-monitoring/node-exporter-zkffp"
Sep 14 09:45:15.969049 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:45:15.968851    2563 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 14 09:45:15.969049 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.968887    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dabf7719-e039-48a1-8150-f100f09e2d4a-node-exporter-wtmp\") pod \"node-exporter-zkffp\" (UID: \"dabf7719-e039-48a1-8150-f100f09e2d4a\") " pod="openshift-monitoring/node-exporter-zkffp"
Sep 14 09:45:15.969049 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:45:15.968929    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dabf7719-e039-48a1-8150-f100f09e2d4a-node-exporter-tls podName:dabf7719-e039-48a1-8150-f100f09e2d4a nodeName:}" failed. No retries permitted until 2026-09-14 09:45:16.468903419 +0000 UTC m=+105.508578029 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/dabf7719-e039-48a1-8150-f100f09e2d4a-node-exporter-tls") pod "node-exporter-zkffp" (UID: "dabf7719-e039-48a1-8150-f100f09e2d4a") : secret "node-exporter-tls" not found
Sep 14 09:45:15.969049 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.968963    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dabf7719-e039-48a1-8150-f100f09e2d4a-node-exporter-accelerators-collector-config\") pod \"node-exporter-zkffp\" (UID: \"dabf7719-e039-48a1-8150-f100f09e2d4a\") " pod="openshift-monitoring/node-exporter-zkffp"
Sep 14 09:45:15.969049 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.969025    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/698195e3-0a13-4176-ab4c-f8167e92eefb-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-ldtq5\" (UID: \"698195e3-0a13-4176-ab4c-f8167e92eefb\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-ldtq5"
Sep 14 09:45:15.969454 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.969064    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/935ba669-ed70-406a-a645-054a8db8b24b-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-v8jjn\" (UID: \"935ba669-ed70-406a-a645-054a8db8b24b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v8jjn"
Sep 14 09:45:15.969454 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.969091    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dabf7719-e039-48a1-8150-f100f09e2d4a-metrics-client-ca\") pod \"node-exporter-zkffp\" (UID: \"dabf7719-e039-48a1-8150-f100f09e2d4a\") " pod="openshift-monitoring/node-exporter-zkffp"
Sep 14 09:45:15.969454 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.969137    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqdbm\" (UniqueName: \"kubernetes.io/projected/698195e3-0a13-4176-ab4c-f8167e92eefb-kube-api-access-tqdbm\") pod \"openshift-state-metrics-67dcf8c979-ldtq5\" (UID: \"698195e3-0a13-4176-ab4c-f8167e92eefb\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-ldtq5"
Sep 14 09:45:15.969454 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.969169    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/698195e3-0a13-4176-ab4c-f8167e92eefb-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-ldtq5\" (UID: \"698195e3-0a13-4176-ab4c-f8167e92eefb\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-ldtq5"
Sep 14 09:45:15.969454 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.969191    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dabf7719-e039-48a1-8150-f100f09e2d4a-node-exporter-textfile\") pod \"node-exporter-zkffp\" (UID: \"dabf7719-e039-48a1-8150-f100f09e2d4a\") " pod="openshift-monitoring/node-exporter-zkffp"
Sep 14 09:45:15.969454 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.969226    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/935ba669-ed70-406a-a645-054a8db8b24b-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-v8jjn\" (UID: \"935ba669-ed70-406a-a645-054a8db8b24b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v8jjn"
Sep 14 09:45:15.969454 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.969251    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/698195e3-0a13-4176-ab4c-f8167e92eefb-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-ldtq5\" (UID: \"698195e3-0a13-4176-ab4c-f8167e92eefb\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-ldtq5"
Sep 14 09:45:15.969454 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.969438    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/935ba669-ed70-406a-a645-054a8db8b24b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-v8jjn\" (UID: \"935ba669-ed70-406a-a645-054a8db8b24b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v8jjn"
Sep 14 09:45:15.969945 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.969618    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dabf7719-e039-48a1-8150-f100f09e2d4a-node-exporter-accelerators-collector-config\") pod \"node-exporter-zkffp\" (UID: \"dabf7719-e039-48a1-8150-f100f09e2d4a\") " pod="openshift-monitoring/node-exporter-zkffp"
Sep 14 09:45:15.969945 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.969711    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/935ba669-ed70-406a-a645-054a8db8b24b-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-v8jjn\" (UID: \"935ba669-ed70-406a-a645-054a8db8b24b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v8jjn"
Sep 14 09:45:15.969945 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.969771    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dabf7719-e039-48a1-8150-f100f09e2d4a-node-exporter-textfile\") pod \"node-exporter-zkffp\" (UID: \"dabf7719-e039-48a1-8150-f100f09e2d4a\") " pod="openshift-monitoring/node-exporter-zkffp"
Sep 14 09:45:15.970213 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.970185    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/698195e3-0a13-4176-ab4c-f8167e92eefb-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-ldtq5\" (UID: \"698195e3-0a13-4176-ab4c-f8167e92eefb\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-ldtq5"
Sep 14 09:45:15.970302 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.970280    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dabf7719-e039-48a1-8150-f100f09e2d4a-metrics-client-ca\") pod \"node-exporter-zkffp\" (UID: \"dabf7719-e039-48a1-8150-f100f09e2d4a\") " pod="openshift-monitoring/node-exporter-zkffp"
Sep 14 09:45:15.970391 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.970367    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/935ba669-ed70-406a-a645-054a8db8b24b-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-v8jjn\" (UID: \"935ba669-ed70-406a-a645-054a8db8b24b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v8jjn"
Sep 14 09:45:15.971587 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.971558    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/698195e3-0a13-4176-ab4c-f8167e92eefb-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-ldtq5\" (UID: \"698195e3-0a13-4176-ab4c-f8167e92eefb\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-ldtq5"
Sep 14 09:45:15.971710 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.971646    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/935ba669-ed70-406a-a645-054a8db8b24b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-v8jjn\" (UID: \"935ba669-ed70-406a-a645-054a8db8b24b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v8jjn"
Sep 14 09:45:15.971953 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.971931    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/698195e3-0a13-4176-ab4c-f8167e92eefb-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-ldtq5\" (UID: \"698195e3-0a13-4176-ab4c-f8167e92eefb\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-ldtq5"
Sep 14 09:45:15.972053 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.972029    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dabf7719-e039-48a1-8150-f100f09e2d4a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zkffp\" (UID: \"dabf7719-e039-48a1-8150-f100f09e2d4a\") " pod="openshift-monitoring/node-exporter-zkffp"
Sep 14 09:45:15.972493 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.972472    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/935ba669-ed70-406a-a645-054a8db8b24b-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-v8jjn\" (UID: \"935ba669-ed70-406a-a645-054a8db8b24b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v8jjn"
Sep 14 09:45:15.985820 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.985782    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rppr\" (UniqueName: \"kubernetes.io/projected/dabf7719-e039-48a1-8150-f100f09e2d4a-kube-api-access-7rppr\") pod \"node-exporter-zkffp\" (UID: \"dabf7719-e039-48a1-8150-f100f09e2d4a\") " pod="openshift-monitoring/node-exporter-zkffp"
Sep 14 09:45:15.986058 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.986034    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mk8c\" (UniqueName: \"kubernetes.io/projected/935ba669-ed70-406a-a645-054a8db8b24b-kube-api-access-9mk8c\") pod \"kube-state-metrics-59fbdd77d-v8jjn\" (UID: \"935ba669-ed70-406a-a645-054a8db8b24b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v8jjn"
Sep 14 09:45:15.986114 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:15.986035    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqdbm\" (UniqueName: \"kubernetes.io/projected/698195e3-0a13-4176-ab4c-f8167e92eefb-kube-api-access-tqdbm\") pod \"openshift-state-metrics-67dcf8c979-ldtq5\" (UID: \"698195e3-0a13-4176-ab4c-f8167e92eefb\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-ldtq5"
Sep 14 09:45:16.014631 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:16.014595    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-ldtq5"
Sep 14 09:45:16.026753 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:16.026721    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v8jjn"
Sep 14 09:45:16.051900 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:16.051856    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:16.150819 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:16.150785    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:16.251659 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:16.251621    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:16.324927 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:16.324897    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-ldtq5"]
Sep 14 09:45:16.327515 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:45:16.327482    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod698195e3_0a13_4176_ab4c_f8167e92eefb.slice/crio-6bb63bbe3c262060958d306894ad02eaba1d00c2b7fcc203c224e85e0e80d7f3 WatchSource:0}: Error finding container 6bb63bbe3c262060958d306894ad02eaba1d00c2b7fcc203c224e85e0e80d7f3: Status 404 returned error can't find the container with id 6bb63bbe3c262060958d306894ad02eaba1d00c2b7fcc203c224e85e0e80d7f3
Sep 14 09:45:16.347847 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:16.347811    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-v8jjn"]
Sep 14 09:45:16.351018 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:45:16.350971    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod935ba669_ed70_406a_a645_054a8db8b24b.slice/crio-066f9c34ee0dcb6723e24d57c0ec77b255111b52f337bc24352786bcfe0131ae WatchSource:0}: Error finding container 066f9c34ee0dcb6723e24d57c0ec77b255111b52f337bc24352786bcfe0131ae: Status 404 returned error can't find the container with id 066f9c34ee0dcb6723e24d57c0ec77b255111b52f337bc24352786bcfe0131ae
Sep 14 09:45:16.408400 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:16.408358    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:16.473996 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:16.473960    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dabf7719-e039-48a1-8150-f100f09e2d4a-node-exporter-tls\") pod \"node-exporter-zkffp\" (UID: \"dabf7719-e039-48a1-8150-f100f09e2d4a\") " pod="openshift-monitoring/node-exporter-zkffp"
Sep 14 09:45:16.476637 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:16.476611    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dabf7719-e039-48a1-8150-f100f09e2d4a-node-exporter-tls\") pod \"node-exporter-zkffp\" (UID: \"dabf7719-e039-48a1-8150-f100f09e2d4a\") " pod="openshift-monitoring/node-exporter-zkffp"
Sep 14 09:45:16.650146 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:16.650104    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zkffp"
Sep 14 09:45:16.653775 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:16.653748    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:16.676536 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:45:16.676506    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddabf7719_e039_48a1_8150_f100f09e2d4a.slice/crio-e88f031b9bba34bc5b05dc3bd49167f47c98b8f4b17127c1eca9af0ea0194d71 WatchSource:0}: Error finding container e88f031b9bba34bc5b05dc3bd49167f47c98b8f4b17127c1eca9af0ea0194d71: Status 404 returned error can't find the container with id e88f031b9bba34bc5b05dc3bd49167f47c98b8f4b17127c1eca9af0ea0194d71
Sep 14 09:45:16.750701 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:16.750661    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:17.059754 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:17.059721    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v8jjn" event={"ID":"935ba669-ed70-406a-a645-054a8db8b24b","Type":"ContainerStarted","Data":"066f9c34ee0dcb6723e24d57c0ec77b255111b52f337bc24352786bcfe0131ae"}
Sep 14 09:45:17.061927 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:17.061894    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-ldtq5" event={"ID":"698195e3-0a13-4176-ab4c-f8167e92eefb","Type":"ContainerStarted","Data":"7c3f7ddc2f861b7d0590653b702ab33fba887bec3eab31ec27cc825f244b9abb"}
Sep 14 09:45:17.062133 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:17.061955    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-ldtq5" event={"ID":"698195e3-0a13-4176-ab4c-f8167e92eefb","Type":"ContainerStarted","Data":"643b1d8a2d1fc752a6bfb1b4832054f5752619650556df02ee957f61135085b6"}
Sep 14 09:45:17.062133 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:17.061985    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-ldtq5" event={"ID":"698195e3-0a13-4176-ab4c-f8167e92eefb","Type":"ContainerStarted","Data":"6bb63bbe3c262060958d306894ad02eaba1d00c2b7fcc203c224e85e0e80d7f3"}
Sep 14 09:45:17.063081 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:17.063056    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zkffp" event={"ID":"dabf7719-e039-48a1-8150-f100f09e2d4a","Type":"ContainerStarted","Data":"e88f031b9bba34bc5b05dc3bd49167f47c98b8f4b17127c1eca9af0ea0194d71"}
Sep 14 09:45:17.300864 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:17.300610    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:17.863324 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:17.863277    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-74f9bbb8b6-nlfpv"
Sep 14 09:45:17.863739 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:17.863394    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-74f9bbb8b6-nlfpv"
Sep 14 09:45:17.865054 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:17.865025    2563 patch_prober.go:28] interesting pod/console-74f9bbb8b6-nlfpv container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.15:8443/health\": dial tcp 10.134.0.15:8443: connect: connection refused" start-of-body=
Sep 14 09:45:17.865201 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:17.865073    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-74f9bbb8b6-nlfpv" podUID="8f9e7a90-aded-4d16-9af6-bde1bccc0002" containerName="console" probeResult="failure" output="Get \"https://10.134.0.15:8443/health\": dial tcp 10.134.0.15:8443: connect: connection refused"
Sep 14 09:45:18.845423 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:18.845323    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:18.947566 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:18.947531    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:19.072878 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:19.072845    2563 generic.go:356] "Generic (PLEG): container finished" podID="dabf7719-e039-48a1-8150-f100f09e2d4a" containerID="7328868839c9f0bff6389c69b9d7f33d1bd45cf9ab52aca0be034822edbc4792" exitCode=0
Sep 14 09:45:19.073090 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:19.072937    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zkffp" event={"ID":"dabf7719-e039-48a1-8150-f100f09e2d4a","Type":"ContainerDied","Data":"7328868839c9f0bff6389c69b9d7f33d1bd45cf9ab52aca0be034822edbc4792"}
Sep 14 09:45:19.075485 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:19.075448    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v8jjn" event={"ID":"935ba669-ed70-406a-a645-054a8db8b24b","Type":"ContainerStarted","Data":"94a0921f47aaec0d760f75ce11ea46d78c41a0039dbac0625eccdb656a49e903"}
Sep 14 09:45:19.075644 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:19.075519    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v8jjn" event={"ID":"935ba669-ed70-406a-a645-054a8db8b24b","Type":"ContainerStarted","Data":"82c4914ca4f200f60d57c1c0002d044469f68f39b2e9a4d457ec5f8ce367646f"}
Sep 14 09:45:19.075644 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:19.075545    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v8jjn" event={"ID":"935ba669-ed70-406a-a645-054a8db8b24b","Type":"ContainerStarted","Data":"428fc32b523d2ec7d820b205995b538a1bbbadec517a8f7f342708a2f75dc0c9"}
Sep 14 09:45:19.077872 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:19.077838    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-ldtq5" event={"ID":"698195e3-0a13-4176-ab4c-f8167e92eefb","Type":"ContainerStarted","Data":"f13574e766a1441face08e509892b1106359c9b9e1f61b5f829e893daa9af5e8"}
Sep 14 09:45:19.117088 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:19.117026    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-ldtq5" podStartSLOduration=2.474127296 podStartE2EDuration="4.11698864s" totalImagesPullingTime="1.642861344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:45:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:45:19.11550608 +0000 UTC m=+108.155180702" watchObservedRunningTime="2026-09-14 09:45:19.11698864 +0000 UTC m=+108.156663260"
Sep 14 09:45:19.151051 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:19.150981    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-v8jjn" podStartSLOduration=2.263517699 podStartE2EDuration="4.150958559s" totalImagesPullingTime="1.88744086s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:45:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:45:19.149591413 +0000 UTC m=+108.189266036" watchObservedRunningTime="2026-09-14 09:45:19.150958559 +0000 UTC m=+108.190633181"
Sep 14 09:45:19.193636 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:19.193601    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:19.292505 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:19.292464    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:19.497529 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:19.497494    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:19.980611 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:19.980527    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-65fb75f47b-lx68z"]
Sep 14 09:45:19.983387 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:19.983362    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-65fb75f47b-lx68z"
Sep 14 09:45:19.986594 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:19.986561    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 14 09:45:19.986760 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:19.986641    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-6etqeaeid7qbs\""
Sep 14 09:45:19.986760 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:19.986743    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 14 09:45:19.986864 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:19.986849    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 14 09:45:19.986971 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:19.986851    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 14 09:45:19.987021 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:19.986958    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-7xqwj\""
Sep 14 09:45:19.995325 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:19.995289    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-65fb75f47b-lx68z"]
Sep 14 09:45:20.086510 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:20.086463    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zkffp" event={"ID":"dabf7719-e039-48a1-8150-f100f09e2d4a","Type":"ContainerStarted","Data":"56c56080a36b469d245688e4d551dcf120cff1a0a30aa78b743996d61bfb5ce0"}
Sep 14 09:45:20.086684 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:20.086570    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zkffp" event={"ID":"dabf7719-e039-48a1-8150-f100f09e2d4a","Type":"ContainerStarted","Data":"5fd1aad77d3db52c01720556c18875fff3703091a31ac261be7dfcc4c868ebef"}
Sep 14 09:45:20.107188 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:20.107130    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-zkffp" podStartSLOduration=3.457154481 podStartE2EDuration="5.107110532s" totalImagesPullingTime="1.559621911s" totalInitContainerRuntime="90.33414ms" isStatefulPod=true podCreationTimestamp="2026-09-14 09:45:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:45:20.105523665 +0000 UTC m=+109.145198286" watchObservedRunningTime="2026-09-14 09:45:20.107110532 +0000 UTC m=+109.146785153"
Sep 14 09:45:20.107595 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:20.107578    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2ecb5cff-e201-48cd-8837-a432af591332-metrics-server-audit-profiles\") pod \"metrics-server-65fb75f47b-lx68z\" (UID: \"2ecb5cff-e201-48cd-8837-a432af591332\") " pod="openshift-monitoring/metrics-server-65fb75f47b-lx68z"
Sep 14 09:45:20.107749 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:20.107624    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2ecb5cff-e201-48cd-8837-a432af591332-secret-metrics-server-tls\") pod \"metrics-server-65fb75f47b-lx68z\" (UID: \"2ecb5cff-e201-48cd-8837-a432af591332\") " pod="openshift-monitoring/metrics-server-65fb75f47b-lx68z"
Sep 14 09:45:20.107749 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:20.107642    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2ecb5cff-e201-48cd-8837-a432af591332-audit-log\") pod \"metrics-server-65fb75f47b-lx68z\" (UID: \"2ecb5cff-e201-48cd-8837-a432af591332\") " pod="openshift-monitoring/metrics-server-65fb75f47b-lx68z"
Sep 14 09:45:20.107749 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:20.107670    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2ecb5cff-e201-48cd-8837-a432af591332-client-ca-bundle\") pod \"metrics-server-65fb75f47b-lx68z\" (UID: \"2ecb5cff-e201-48cd-8837-a432af591332\") " pod="openshift-monitoring/metrics-server-65fb75f47b-lx68z"
Sep 14 09:45:20.107749 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:20.107696    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ecb5cff-e201-48cd-8837-a432af591332-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-65fb75f47b-lx68z\" (UID: \"2ecb5cff-e201-48cd-8837-a432af591332\") " pod="openshift-monitoring/metrics-server-65fb75f47b-lx68z"
Sep 14 09:45:20.107749 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:20.107717    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx72q\" (UniqueName: \"kubernetes.io/projected/2ecb5cff-e201-48cd-8837-a432af591332-kube-api-access-rx72q\") pod \"metrics-server-65fb75f47b-lx68z\" (UID: \"2ecb5cff-e201-48cd-8837-a432af591332\") " pod="openshift-monitoring/metrics-server-65fb75f47b-lx68z"
Sep 14 09:45:20.107749 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:20.107741    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2ecb5cff-e201-48cd-8837-a432af591332-secret-metrics-server-client-certs\") pod \"metrics-server-65fb75f47b-lx68z\" (UID: \"2ecb5cff-e201-48cd-8837-a432af591332\") " pod="openshift-monitoring/metrics-server-65fb75f47b-lx68z"
Sep 14 09:45:20.208647 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:20.208604    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2ecb5cff-e201-48cd-8837-a432af591332-secret-metrics-server-tls\") pod \"metrics-server-65fb75f47b-lx68z\" (UID: \"2ecb5cff-e201-48cd-8837-a432af591332\") " pod="openshift-monitoring/metrics-server-65fb75f47b-lx68z"
Sep 14 09:45:20.208831 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:20.208682    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2ecb5cff-e201-48cd-8837-a432af591332-audit-log\") pod \"metrics-server-65fb75f47b-lx68z\" (UID: \"2ecb5cff-e201-48cd-8837-a432af591332\") " pod="openshift-monitoring/metrics-server-65fb75f47b-lx68z"
Sep 14 09:45:20.209323 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:20.209034    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2ecb5cff-e201-48cd-8837-a432af591332-client-ca-bundle\") pod \"metrics-server-65fb75f47b-lx68z\" (UID: \"2ecb5cff-e201-48cd-8837-a432af591332\") " pod="openshift-monitoring/metrics-server-65fb75f47b-lx68z"
Sep 14 09:45:20.209323 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:20.209105    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ecb5cff-e201-48cd-8837-a432af591332-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-65fb75f47b-lx68z\" (UID: \"2ecb5cff-e201-48cd-8837-a432af591332\") " pod="openshift-monitoring/metrics-server-65fb75f47b-lx68z"
Sep 14 09:45:20.209323 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:20.209142    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2ecb5cff-e201-48cd-8837-a432af591332-audit-log\") pod \"metrics-server-65fb75f47b-lx68z\" (UID: \"2ecb5cff-e201-48cd-8837-a432af591332\") " pod="openshift-monitoring/metrics-server-65fb75f47b-lx68z"
Sep 14 09:45:20.209323 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:20.209152    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx72q\" (UniqueName: \"kubernetes.io/projected/2ecb5cff-e201-48cd-8837-a432af591332-kube-api-access-rx72q\") pod \"metrics-server-65fb75f47b-lx68z\" (UID: \"2ecb5cff-e201-48cd-8837-a432af591332\") " pod="openshift-monitoring/metrics-server-65fb75f47b-lx68z"
Sep 14 09:45:20.209323 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:20.209266    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2ecb5cff-e201-48cd-8837-a432af591332-secret-metrics-server-client-certs\") pod \"metrics-server-65fb75f47b-lx68z\" (UID: \"2ecb5cff-e201-48cd-8837-a432af591332\") " pod="openshift-monitoring/metrics-server-65fb75f47b-lx68z"
Sep 14 09:45:20.209766 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:20.209377    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2ecb5cff-e201-48cd-8837-a432af591332-metrics-server-audit-profiles\") pod \"metrics-server-65fb75f47b-lx68z\" (UID: \"2ecb5cff-e201-48cd-8837-a432af591332\") " pod="openshift-monitoring/metrics-server-65fb75f47b-lx68z"
Sep 14 09:45:20.209828 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:20.209771    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ecb5cff-e201-48cd-8837-a432af591332-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-65fb75f47b-lx68z\" (UID: \"2ecb5cff-e201-48cd-8837-a432af591332\") " pod="openshift-monitoring/metrics-server-65fb75f47b-lx68z"
Sep 14 09:45:20.210366 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:20.210342    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2ecb5cff-e201-48cd-8837-a432af591332-metrics-server-audit-profiles\") pod \"metrics-server-65fb75f47b-lx68z\" (UID: \"2ecb5cff-e201-48cd-8837-a432af591332\") " pod="openshift-monitoring/metrics-server-65fb75f47b-lx68z"
Sep 14 09:45:20.211404 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:20.211374    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2ecb5cff-e201-48cd-8837-a432af591332-secret-metrics-server-tls\") pod \"metrics-server-65fb75f47b-lx68z\" (UID: \"2ecb5cff-e201-48cd-8837-a432af591332\") " pod="openshift-monitoring/metrics-server-65fb75f47b-lx68z"
Sep 14 09:45:20.211482 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:20.211400    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2ecb5cff-e201-48cd-8837-a432af591332-client-ca-bundle\") pod \"metrics-server-65fb75f47b-lx68z\" (UID: \"2ecb5cff-e201-48cd-8837-a432af591332\") " pod="openshift-monitoring/metrics-server-65fb75f47b-lx68z"
Sep 14 09:45:20.211803 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:20.211784    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2ecb5cff-e201-48cd-8837-a432af591332-secret-metrics-server-client-certs\") pod \"metrics-server-65fb75f47b-lx68z\" (UID: \"2ecb5cff-e201-48cd-8837-a432af591332\") " pod="openshift-monitoring/metrics-server-65fb75f47b-lx68z"
Sep 14 09:45:20.217909 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:20.217874    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx72q\" (UniqueName: \"kubernetes.io/projected/2ecb5cff-e201-48cd-8837-a432af591332-kube-api-access-rx72q\") pod \"metrics-server-65fb75f47b-lx68z\" (UID: \"2ecb5cff-e201-48cd-8837-a432af591332\") " pod="openshift-monitoring/metrics-server-65fb75f47b-lx68z"
Sep 14 09:45:20.295516 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:20.295477    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-65fb75f47b-lx68z"
Sep 14 09:45:20.416127 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:20.416085    2563 patch_prober.go:28] interesting pod/image-registry-9689ffdd5-8254l container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 14 09:45:20.416300 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:20.416150    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-9689ffdd5-8254l" podUID="cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 14 09:45:20.465644 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:20.465605    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-n75c8"]
Sep 14 09:45:20.468234 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:20.468210    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-n75c8"
Sep 14 09:45:20.470879 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:20.470847    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 14 09:45:20.470879 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:20.470866    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-62lcr\""
Sep 14 09:45:20.478678 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:20.478649    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-n75c8"]
Sep 14 09:45:20.553973 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:20.553888    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:20.577479 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:20.577426    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-65fb75f47b-lx68z"]
Sep 14 09:45:20.579700 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:45:20.579670    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ecb5cff_e201_48cd_8837_a432af591332.slice/crio-d0907a1d5fdcfa8ef5d4411e9e82ace66f2400542dbb3b1e21fd2e3b9b5c8849 WatchSource:0}: Error finding container d0907a1d5fdcfa8ef5d4411e9e82ace66f2400542dbb3b1e21fd2e3b9b5c8849: Status 404 returned error can't find the container with id d0907a1d5fdcfa8ef5d4411e9e82ace66f2400542dbb3b1e21fd2e3b9b5c8849
Sep 14 09:45:20.612632 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:20.612592    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw2jz\" (UniqueName: \"kubernetes.io/projected/81e3b875-1204-4d99-b431-8ded39ac08e8-kube-api-access-tw2jz\") pod \"monitoring-plugin-5d85778b9d-n75c8\" (UID: \"81e3b875-1204-4d99-b431-8ded39ac08e8\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-n75c8"
Sep 14 09:45:20.612829 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:20.612709    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/81e3b875-1204-4d99-b431-8ded39ac08e8-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-n75c8\" (UID: \"81e3b875-1204-4d99-b431-8ded39ac08e8\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-n75c8"
Sep 14 09:45:20.691366 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:20.691319    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:20.713280 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:20.713238    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/81e3b875-1204-4d99-b431-8ded39ac08e8-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-n75c8\" (UID: \"81e3b875-1204-4d99-b431-8ded39ac08e8\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-n75c8"
Sep 14 09:45:20.713447 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:20.713320    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw2jz\" (UniqueName: \"kubernetes.io/projected/81e3b875-1204-4d99-b431-8ded39ac08e8-kube-api-access-tw2jz\") pod \"monitoring-plugin-5d85778b9d-n75c8\" (UID: \"81e3b875-1204-4d99-b431-8ded39ac08e8\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-n75c8"
Sep 14 09:45:20.715820 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:20.715795    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/81e3b875-1204-4d99-b431-8ded39ac08e8-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-n75c8\" (UID: \"81e3b875-1204-4d99-b431-8ded39ac08e8\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-n75c8"
Sep 14 09:45:20.721580 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:20.721553    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw2jz\" (UniqueName: \"kubernetes.io/projected/81e3b875-1204-4d99-b431-8ded39ac08e8-kube-api-access-tw2jz\") pod \"monitoring-plugin-5d85778b9d-n75c8\" (UID: \"81e3b875-1204-4d99-b431-8ded39ac08e8\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-n75c8"
Sep 14 09:45:20.776259 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:20.776220    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:20.779137 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:20.779096    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-n75c8"
Sep 14 09:45:20.831090 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:20.830995    2563 patch_prober.go:28] interesting pod/image-registry-544f748fdb-5pr6x container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 14 09:45:20.832145 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:20.831109    2563 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-544f748fdb-5pr6x" podUID="f610700e-43a1-443a-a589-3420e9ce4c91" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 14 09:45:21.053601 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:21.053568    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-n75c8"]
Sep 14 09:45:21.056479 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:45:21.056447    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81e3b875_1204_4d99_b431_8ded39ac08e8.slice/crio-dd9e61bd3026633666ed15e799196b0ccbf4c7832bd6763a5f79fd930f95fff6 WatchSource:0}: Error finding container dd9e61bd3026633666ed15e799196b0ccbf4c7832bd6763a5f79fd930f95fff6: Status 404 returned error can't find the container with id dd9e61bd3026633666ed15e799196b0ccbf4c7832bd6763a5f79fd930f95fff6
Sep 14 09:45:21.091724 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:21.091614    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-65fb75f47b-lx68z" event={"ID":"2ecb5cff-e201-48cd-8837-a432af591332","Type":"ContainerStarted","Data":"d0907a1d5fdcfa8ef5d4411e9e82ace66f2400542dbb3b1e21fd2e3b9b5c8849"}
Sep 14 09:45:21.092898 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:21.092865    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-n75c8" event={"ID":"81e3b875-1204-4d99-b431-8ded39ac08e8","Type":"ContainerStarted","Data":"dd9e61bd3026633666ed15e799196b0ccbf4c7832bd6763a5f79fd930f95fff6"}
Sep 14 09:45:22.147623 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:22.147578    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:23.001183 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:23.001141    2563 patch_prober.go:28] interesting pod/image-registry-544f748fdb-5pr6x container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 14 09:45:23.001362 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:23.001213    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-544f748fdb-5pr6x" podUID="f610700e-43a1-443a-a589-3420e9ce4c91" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 14 09:45:23.050301 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:23.050262    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-mklrw"
Sep 14 09:45:23.103313 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:23.103276    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-n75c8" event={"ID":"81e3b875-1204-4d99-b431-8ded39ac08e8","Type":"ContainerStarted","Data":"8edbe9c88c8a3bba8af5563c4c60b8e7ba00fa8e60c5a28fd674fc04374df906"}
Sep 14 09:45:23.103558 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:23.103533    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-n75c8"
Sep 14 09:45:23.105349 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:23.105305    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-65fb75f47b-lx68z" event={"ID":"2ecb5cff-e201-48cd-8837-a432af591332","Type":"ContainerStarted","Data":"e87ce7d548f41ab681de141c0701ee1da4990326c8465f45bf7186c825e2f4c4"}
Sep 14 09:45:23.108686 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:23.108649    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-n75c8"
Sep 14 09:45:23.120320 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:23.120261    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-n75c8" podStartSLOduration=1.698130962 podStartE2EDuration="3.120242994s" totalImagesPullingTime="1.422112032s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:45:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:45:23.119778024 +0000 UTC m=+112.159452644" watchObservedRunningTime="2026-09-14 09:45:23.120242994 +0000 UTC m=+112.159917615"
Sep 14 09:45:23.138018 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:23.137937    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-65fb75f47b-lx68z" podStartSLOduration=2.241926961 podStartE2EDuration="4.137916212s" totalImagesPullingTime="1.895989251s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:45:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:45:23.136933981 +0000 UTC m=+112.176608602" watchObservedRunningTime="2026-09-14 09:45:23.137916212 +0000 UTC m=+112.177590850"
Sep 14 09:45:23.576280 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:23.576249    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:23.640644 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:23.640603    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:25.432288 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:25.432244    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-9689ffdd5-8254l" podUID="cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8" containerName="registry" containerID="cri-o://5060c5e4d3132cbe99409e76608c33b243b952cb3c684e4c06b665fd117b912d" gracePeriod=30
Sep 14 09:45:25.678169 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:25.678134    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-9689ffdd5-8254l"
Sep 14 09:45:25.753965 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:25.753932    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8-registry-certificates\") pod \"cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8\" (UID: \"cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8\") "
Sep 14 09:45:25.754165 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:25.753974    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cs782\" (UniqueName: \"kubernetes.io/projected/cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8-kube-api-access-cs782\") pod \"cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8\" (UID: \"cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8\") "
Sep 14 09:45:25.754165 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:25.754018    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8-registry-tls\") pod \"cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8\" (UID: \"cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8\") "
Sep 14 09:45:25.754165 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:25.754039    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8-image-registry-private-configuration\") pod \"cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8\" (UID: \"cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8\") "
Sep 14 09:45:25.754165 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:25.754064    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8-trusted-ca\") pod \"cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8\" (UID: \"cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8\") "
Sep 14 09:45:25.754165 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:25.754124    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8-installation-pull-secrets\") pod \"cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8\" (UID: \"cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8\") "
Sep 14 09:45:25.754417 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:25.754175    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8-ca-trust-extracted\") pod \"cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8\" (UID: \"cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8\") "
Sep 14 09:45:25.754417 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:25.754217    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8-bound-sa-token\") pod \"cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8\" (UID: \"cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8\") "
Sep 14 09:45:25.754586 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:25.754554    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8" (UID: "cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 09:45:25.754996 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:25.754965    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8" (UID: "cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 09:45:25.756612 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:25.756574    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8" (UID: "cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 09:45:25.756821 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:25.756797    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8-kube-api-access-cs782" (OuterVolumeSpecName: "kube-api-access-cs782") pod "cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8" (UID: "cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8"). InnerVolumeSpecName "kube-api-access-cs782". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 09:45:25.756942 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:25.756841    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8" (UID: "cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 09:45:25.757029 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:25.756936    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8" (UID: "cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 09:45:25.757029 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:25.756950    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8" (UID: "cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 09:45:25.762997 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:25.762964    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8" (UID: "cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 09:45:25.855724 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:25.855685    2563 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8-installation-pull-secrets\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:45:25.855724 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:25.855715    2563 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8-ca-trust-extracted\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:45:25.855724 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:25.855726    2563 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8-bound-sa-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:45:25.855724 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:25.855735    2563 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8-registry-certificates\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:45:25.855972 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:25.855744    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cs782\" (UniqueName: \"kubernetes.io/projected/cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8-kube-api-access-cs782\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:45:25.855972 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:25.855752    2563 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8-registry-tls\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:45:25.855972 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:25.855761    2563 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8-image-registry-private-configuration\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:45:25.855972 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:25.855770    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8-trusted-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:45:26.118914 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:26.118808    2563 generic.go:356] "Generic (PLEG): container finished" podID="cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8" containerID="5060c5e4d3132cbe99409e76608c33b243b952cb3c684e4c06b665fd117b912d" exitCode=0
Sep 14 09:45:26.118914 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:26.118858    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-9689ffdd5-8254l"
Sep 14 09:45:26.118914 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:26.118891    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-9689ffdd5-8254l" event={"ID":"cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8","Type":"ContainerDied","Data":"5060c5e4d3132cbe99409e76608c33b243b952cb3c684e4c06b665fd117b912d"}
Sep 14 09:45:26.119220 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:26.118945    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-9689ffdd5-8254l" event={"ID":"cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8","Type":"ContainerDied","Data":"c96b1e4306b0fe1fa6b320205ed92126a13ce01819d4f230298c483616ee7dcb"}
Sep 14 09:45:26.119220 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:26.118967    2563 scope.go:117] "RemoveContainer" containerID="5060c5e4d3132cbe99409e76608c33b243b952cb3c684e4c06b665fd117b912d"
Sep 14 09:45:26.131326 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:26.131300    2563 scope.go:117] "RemoveContainer" containerID="5060c5e4d3132cbe99409e76608c33b243b952cb3c684e4c06b665fd117b912d"
Sep 14 09:45:26.131769 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:45:26.131733    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5060c5e4d3132cbe99409e76608c33b243b952cb3c684e4c06b665fd117b912d\": container with ID starting with 5060c5e4d3132cbe99409e76608c33b243b952cb3c684e4c06b665fd117b912d not found: ID does not exist" containerID="5060c5e4d3132cbe99409e76608c33b243b952cb3c684e4c06b665fd117b912d"
Sep 14 09:45:26.131872 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:26.131776    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5060c5e4d3132cbe99409e76608c33b243b952cb3c684e4c06b665fd117b912d"} err="failed to get container status \"5060c5e4d3132cbe99409e76608c33b243b952cb3c684e4c06b665fd117b912d\": rpc error: code = NotFound desc = could not find container \"5060c5e4d3132cbe99409e76608c33b243b952cb3c684e4c06b665fd117b912d\": container with ID starting with 5060c5e4d3132cbe99409e76608c33b243b952cb3c684e4c06b665fd117b912d not found: ID does not exist"
Sep 14 09:45:26.143699 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:26.143660    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-9689ffdd5-8254l"]
Sep 14 09:45:26.152042 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:26.151982    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-9689ffdd5-8254l"]
Sep 14 09:45:26.170504 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:26.170473    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:26.271702 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:26.271633    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:27.480892 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:27.480853    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8" path="/var/lib/kubelet/pods/cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8/volumes"
Sep 14 09:45:27.867464 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:27.867432    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-74f9bbb8b6-nlfpv"
Sep 14 09:45:27.871538 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:27.871500    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-74f9bbb8b6-nlfpv"
Sep 14 09:45:30.382068 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:30.382024    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:30.829755 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:30.829720    2563 patch_prober.go:28] interesting pod/image-registry-544f748fdb-5pr6x container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 14 09:45:30.829950 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:30.829787    2563 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-544f748fdb-5pr6x" podUID="f610700e-43a1-443a-a589-3420e9ce4c91" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 14 09:45:31.757383 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:31.757341    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:31.830635 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:31.830595    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:32.281250 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:32.281217    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:32.354528 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:32.354492    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:32.713391 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:32.713306    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:32.792711 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:32.792681    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:33.001200 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:33.001173    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-544f748fdb-5pr6x"
Sep 14 09:45:33.216637 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:33.216592    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:33.313797 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:33.313707    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:34.167667 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:34.167629    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:40.296293 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:40.296249    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-65fb75f47b-lx68z"
Sep 14 09:45:40.296900 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:40.296365    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-65fb75f47b-lx68z"
Sep 14 09:45:41.295887 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:41.295846    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d956c018-b9cb-4c19-8f08-16652611d8c3-metrics-certs\") pod \"network-metrics-daemon-6l89k\" (UID: \"d956c018-b9cb-4c19-8f08-16652611d8c3\") " pod="openshift-multus/network-metrics-daemon-6l89k"
Sep 14 09:45:41.298341 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:41.298307    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d956c018-b9cb-4c19-8f08-16652611d8c3-metrics-certs\") pod \"network-metrics-daemon-6l89k\" (UID: \"d956c018-b9cb-4c19-8f08-16652611d8c3\") " pod="openshift-multus/network-metrics-daemon-6l89k"
Sep 14 09:45:41.378068 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:41.378023    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74f9bbb8b6-nlfpv"]
Sep 14 09:45:41.492726 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:41.492686    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:41.590611 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:41.590516    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:41.596223 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:41.596181    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9ctbn\""
Sep 14 09:45:41.604105 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:41.604071    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6l89k"
Sep 14 09:45:41.653427 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:41.653390    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:41.786868 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:41.786827    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:41.885064 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:41.884955    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6l89k"]
Sep 14 09:45:41.887925 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:41.887886    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:41.888679 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:45:41.888652    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd956c018_b9cb_4c19_8f08_16652611d8c3.slice/crio-ec1e14186cabdf241faab1d3d8f25e21bb27c8b18418a0a522b8f3330476a8a3 WatchSource:0}: Error finding container ec1e14186cabdf241faab1d3d8f25e21bb27c8b18418a0a522b8f3330476a8a3: Status 404 returned error can't find the container with id ec1e14186cabdf241faab1d3d8f25e21bb27c8b18418a0a522b8f3330476a8a3
Sep 14 09:45:42.180246 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:42.180149    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6l89k" event={"ID":"d956c018-b9cb-4c19-8f08-16652611d8c3","Type":"ContainerStarted","Data":"ec1e14186cabdf241faab1d3d8f25e21bb27c8b18418a0a522b8f3330476a8a3"}
Sep 14 09:45:43.185187 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:43.185156    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6l89k" event={"ID":"d956c018-b9cb-4c19-8f08-16652611d8c3","Type":"ContainerStarted","Data":"101ab87827a6dc4b5ac1ae3084159d202055ad0996c6140c823c0fd2424c82a9"}
Sep 14 09:45:44.194211 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:44.194172    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6l89k" event={"ID":"d956c018-b9cb-4c19-8f08-16652611d8c3","Type":"ContainerStarted","Data":"11ec80d75e571d176e39f48b7dffdc79250041b5567f0153b61c73847af76ad8"}
Sep 14 09:45:44.212788 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:44.212725    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-6l89k" podStartSLOduration=132.078876774 podStartE2EDuration="2m13.212702583s" totalImagesPullingTime="1.133825809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:43:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:45:44.211903501 +0000 UTC m=+133.251578112" watchObservedRunningTime="2026-09-14 09:45:44.212702583 +0000 UTC m=+133.252377203"
Sep 14 09:45:44.305953 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:44.305919    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:44.442118 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:44.442082    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:46.506510 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:46.506476    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:46.611680 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:46.611645    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:48.929452 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:48.929416    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:49.029982 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:49.029945    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:49.132134 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:49.132097    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:55.548746 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:55.548706    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:45:55.650147 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:45:55.650109    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:46:00.301825 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:00.301790    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-65fb75f47b-lx68z"
Sep 14 09:46:00.306127 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:00.306098    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-65fb75f47b-lx68z"
Sep 14 09:46:01.359651 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:01.359611    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:46:06.214313 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:06.214280    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mklrw_ec1015e1-7070-4f02-8f4e-2fd5d395c4e9/dns/0.log"
Sep 14 09:46:06.221471 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:06.221442    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mklrw_ec1015e1-7070-4f02-8f4e-2fd5d395c4e9/kube-rbac-proxy/0.log"
Sep 14 09:46:06.402094 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:06.402046    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-74f9bbb8b6-nlfpv" podUID="8f9e7a90-aded-4d16-9af6-bde1bccc0002" containerName="console" containerID="cri-o://00e232da119e791f26180bcd91ca511edc526cb8d840994dbd1cc3e6c90b1f90" gracePeriod=15
Sep 14 09:46:06.584147 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:06.584117    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-f5jts_da4b31da-fcbb-4132-bc49-1ecd91bb65bb/dns-node-resolver/0.log"
Sep 14 09:46:06.650602 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:06.650563    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74f9bbb8b6-nlfpv"
Sep 14 09:46:06.810614 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:06.810518    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f9e7a90-aded-4d16-9af6-bde1bccc0002-trusted-ca-bundle\") pod \"8f9e7a90-aded-4d16-9af6-bde1bccc0002\" (UID: \"8f9e7a90-aded-4d16-9af6-bde1bccc0002\") "
Sep 14 09:46:06.810614 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:06.810566    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f9e7a90-aded-4d16-9af6-bde1bccc0002-console-serving-cert\") pod \"8f9e7a90-aded-4d16-9af6-bde1bccc0002\" (UID: \"8f9e7a90-aded-4d16-9af6-bde1bccc0002\") "
Sep 14 09:46:06.810614 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:06.810583    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-44mq2\" (UniqueName: \"kubernetes.io/projected/8f9e7a90-aded-4d16-9af6-bde1bccc0002-kube-api-access-44mq2\") pod \"8f9e7a90-aded-4d16-9af6-bde1bccc0002\" (UID: \"8f9e7a90-aded-4d16-9af6-bde1bccc0002\") "
Sep 14 09:46:06.810850 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:06.810640    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f9e7a90-aded-4d16-9af6-bde1bccc0002-console-oauth-config\") pod \"8f9e7a90-aded-4d16-9af6-bde1bccc0002\" (UID: \"8f9e7a90-aded-4d16-9af6-bde1bccc0002\") "
Sep 14 09:46:06.810850 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:06.810709    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f9e7a90-aded-4d16-9af6-bde1bccc0002-service-ca\") pod \"8f9e7a90-aded-4d16-9af6-bde1bccc0002\" (UID: \"8f9e7a90-aded-4d16-9af6-bde1bccc0002\") "
Sep 14 09:46:06.810850 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:06.810753    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f9e7a90-aded-4d16-9af6-bde1bccc0002-oauth-serving-cert\") pod \"8f9e7a90-aded-4d16-9af6-bde1bccc0002\" (UID: \"8f9e7a90-aded-4d16-9af6-bde1bccc0002\") "
Sep 14 09:46:06.810850 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:06.810783    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8f9e7a90-aded-4d16-9af6-bde1bccc0002-console-config\") pod \"8f9e7a90-aded-4d16-9af6-bde1bccc0002\" (UID: \"8f9e7a90-aded-4d16-9af6-bde1bccc0002\") "
Sep 14 09:46:06.811167 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:06.810983    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f9e7a90-aded-4d16-9af6-bde1bccc0002-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "8f9e7a90-aded-4d16-9af6-bde1bccc0002" (UID: "8f9e7a90-aded-4d16-9af6-bde1bccc0002"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 09:46:06.811263 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:06.811151    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f9e7a90-aded-4d16-9af6-bde1bccc0002-service-ca" (OuterVolumeSpecName: "service-ca") pod "8f9e7a90-aded-4d16-9af6-bde1bccc0002" (UID: "8f9e7a90-aded-4d16-9af6-bde1bccc0002"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 09:46:06.811263 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:06.811243    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f9e7a90-aded-4d16-9af6-bde1bccc0002-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8f9e7a90-aded-4d16-9af6-bde1bccc0002" (UID: "8f9e7a90-aded-4d16-9af6-bde1bccc0002"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 09:46:06.811349 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:06.811283    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f9e7a90-aded-4d16-9af6-bde1bccc0002-console-config" (OuterVolumeSpecName: "console-config") pod "8f9e7a90-aded-4d16-9af6-bde1bccc0002" (UID: "8f9e7a90-aded-4d16-9af6-bde1bccc0002"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 09:46:06.813098 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:06.813069    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f9e7a90-aded-4d16-9af6-bde1bccc0002-kube-api-access-44mq2" (OuterVolumeSpecName: "kube-api-access-44mq2") pod "8f9e7a90-aded-4d16-9af6-bde1bccc0002" (UID: "8f9e7a90-aded-4d16-9af6-bde1bccc0002"). InnerVolumeSpecName "kube-api-access-44mq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 09:46:06.813347 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:06.813327    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f9e7a90-aded-4d16-9af6-bde1bccc0002-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8f9e7a90-aded-4d16-9af6-bde1bccc0002" (UID: "8f9e7a90-aded-4d16-9af6-bde1bccc0002"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 09:46:06.813406 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:06.813356    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f9e7a90-aded-4d16-9af6-bde1bccc0002-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8f9e7a90-aded-4d16-9af6-bde1bccc0002" (UID: "8f9e7a90-aded-4d16-9af6-bde1bccc0002"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 09:46:06.911690 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:06.911647    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f9e7a90-aded-4d16-9af6-bde1bccc0002-console-serving-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:46:06.911690 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:06.911683    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44mq2\" (UniqueName: \"kubernetes.io/projected/8f9e7a90-aded-4d16-9af6-bde1bccc0002-kube-api-access-44mq2\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:46:06.911690 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:06.911692    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f9e7a90-aded-4d16-9af6-bde1bccc0002-console-oauth-config\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:46:06.911690 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:06.911702    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f9e7a90-aded-4d16-9af6-bde1bccc0002-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:46:06.911966 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:06.911711    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f9e7a90-aded-4d16-9af6-bde1bccc0002-oauth-serving-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:46:06.911966 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:06.911720    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8f9e7a90-aded-4d16-9af6-bde1bccc0002-console-config\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:46:06.911966 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:06.911728    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f9e7a90-aded-4d16-9af6-bde1bccc0002-trusted-ca-bundle\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:46:07.287025 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:07.286960    2563 generic.go:356] "Generic (PLEG): container finished" podID="8f9e7a90-aded-4d16-9af6-bde1bccc0002" containerID="00e232da119e791f26180bcd91ca511edc526cb8d840994dbd1cc3e6c90b1f90" exitCode=0
Sep 14 09:46:07.287470 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:07.287044    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74f9bbb8b6-nlfpv"
Sep 14 09:46:07.287470 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:07.287041    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74f9bbb8b6-nlfpv" event={"ID":"8f9e7a90-aded-4d16-9af6-bde1bccc0002","Type":"ContainerDied","Data":"00e232da119e791f26180bcd91ca511edc526cb8d840994dbd1cc3e6c90b1f90"}
Sep 14 09:46:07.287470 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:07.287126    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74f9bbb8b6-nlfpv" event={"ID":"8f9e7a90-aded-4d16-9af6-bde1bccc0002","Type":"ContainerDied","Data":"9a7b7dfb8bcfdc7841032d2702b2c986ce8522fe3969913a7d82731c22f461e3"}
Sep 14 09:46:07.287470 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:07.287149    2563 scope.go:117] "RemoveContainer" containerID="00e232da119e791f26180bcd91ca511edc526cb8d840994dbd1cc3e6c90b1f90"
Sep 14 09:46:07.296214 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:07.296189    2563 scope.go:117] "RemoveContainer" containerID="00e232da119e791f26180bcd91ca511edc526cb8d840994dbd1cc3e6c90b1f90"
Sep 14 09:46:07.296570 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:46:07.296545    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00e232da119e791f26180bcd91ca511edc526cb8d840994dbd1cc3e6c90b1f90\": container with ID starting with 00e232da119e791f26180bcd91ca511edc526cb8d840994dbd1cc3e6c90b1f90 not found: ID does not exist" containerID="00e232da119e791f26180bcd91ca511edc526cb8d840994dbd1cc3e6c90b1f90"
Sep 14 09:46:07.296649 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:07.296581    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00e232da119e791f26180bcd91ca511edc526cb8d840994dbd1cc3e6c90b1f90"} err="failed to get container status \"00e232da119e791f26180bcd91ca511edc526cb8d840994dbd1cc3e6c90b1f90\": rpc error: code = NotFound desc = could not find container \"00e232da119e791f26180bcd91ca511edc526cb8d840994dbd1cc3e6c90b1f90\": container with ID starting with 00e232da119e791f26180bcd91ca511edc526cb8d840994dbd1cc3e6c90b1f90 not found: ID does not exist"
Sep 14 09:46:07.308609 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:07.308571    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74f9bbb8b6-nlfpv"]
Sep 14 09:46:07.312084 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:07.311994    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-74f9bbb8b6-nlfpv"]
Sep 14 09:46:07.481441 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:07.481404    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f9e7a90-aded-4d16-9af6-bde1bccc0002" path="/var/lib/kubelet/pods/8f9e7a90-aded-4d16-9af6-bde1bccc0002/volumes"
Sep 14 09:46:09.530553 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:09.530518    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:46:18.557141 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:18.557093    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:46:18.660704 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:18.660667    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:46:32.032044 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:32.031989    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:46:32.125057 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:32.125026    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:46:43.955080 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:43.955043    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:46:44.013819 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:44.013761    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:46:46.494583 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:46.494542    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:46:46.592858 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:46.592816    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:46:49.136055 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:49.136016    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-57946b7674-fjvqx"]
Sep 14 09:46:49.136445 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:49.136300    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f9e7a90-aded-4d16-9af6-bde1bccc0002" containerName="console"
Sep 14 09:46:49.136445 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:49.136311    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f9e7a90-aded-4d16-9af6-bde1bccc0002" containerName="console"
Sep 14 09:46:49.136445 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:49.136322    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8" containerName="registry"
Sep 14 09:46:49.136445 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:49.136327    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8" containerName="registry"
Sep 14 09:46:49.136445 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:49.136381    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd1d6690-5f55-428e-b4e2-64c3a2a7b4d8" containerName="registry"
Sep 14 09:46:49.136445 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:49.136389    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f9e7a90-aded-4d16-9af6-bde1bccc0002" containerName="console"
Sep 14 09:46:49.140628 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:49.140600    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57946b7674-fjvqx"
Sep 14 09:46:49.143645 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:49.143618    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-stffz\""
Sep 14 09:46:49.143836 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:49.143619    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 14 09:46:49.144197 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:49.144175    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 14 09:46:49.144301 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:49.144282    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 14 09:46:49.144366 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:49.144299    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 14 09:46:49.144837 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:49.144823    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 14 09:46:49.144925 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:49.144910    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 14 09:46:49.144985 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:49.144974    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 14 09:46:49.149223 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:49.149195    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 14 09:46:49.158604 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:49.158580    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57946b7674-fjvqx"]
Sep 14 09:46:49.186577 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:49.186537    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:46:49.226301 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:49.226266    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lqgv\" (UniqueName: \"kubernetes.io/projected/55004416-09dd-42f4-a4e3-520c3e7a0094-kube-api-access-4lqgv\") pod \"console-57946b7674-fjvqx\" (UID: \"55004416-09dd-42f4-a4e3-520c3e7a0094\") " pod="openshift-console/console-57946b7674-fjvqx"
Sep 14 09:46:49.226301 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:49.226303    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/55004416-09dd-42f4-a4e3-520c3e7a0094-service-ca\") pod \"console-57946b7674-fjvqx\" (UID: \"55004416-09dd-42f4-a4e3-520c3e7a0094\") " pod="openshift-console/console-57946b7674-fjvqx"
Sep 14 09:46:49.226536 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:49.226322    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/55004416-09dd-42f4-a4e3-520c3e7a0094-oauth-serving-cert\") pod \"console-57946b7674-fjvqx\" (UID: \"55004416-09dd-42f4-a4e3-520c3e7a0094\") " pod="openshift-console/console-57946b7674-fjvqx"
Sep 14 09:46:49.226536 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:49.226357    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/55004416-09dd-42f4-a4e3-520c3e7a0094-console-serving-cert\") pod \"console-57946b7674-fjvqx\" (UID: \"55004416-09dd-42f4-a4e3-520c3e7a0094\") " pod="openshift-console/console-57946b7674-fjvqx"
Sep 14 09:46:49.226536 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:49.226376    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55004416-09dd-42f4-a4e3-520c3e7a0094-trusted-ca-bundle\") pod \"console-57946b7674-fjvqx\" (UID: \"55004416-09dd-42f4-a4e3-520c3e7a0094\") " pod="openshift-console/console-57946b7674-fjvqx"
Sep 14 09:46:49.226536 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:49.226405    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/55004416-09dd-42f4-a4e3-520c3e7a0094-console-oauth-config\") pod \"console-57946b7674-fjvqx\" (UID: \"55004416-09dd-42f4-a4e3-520c3e7a0094\") " pod="openshift-console/console-57946b7674-fjvqx"
Sep 14 09:46:49.226536 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:49.226486    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/55004416-09dd-42f4-a4e3-520c3e7a0094-console-config\") pod \"console-57946b7674-fjvqx\" (UID: \"55004416-09dd-42f4-a4e3-520c3e7a0094\") " pod="openshift-console/console-57946b7674-fjvqx"
Sep 14 09:46:49.254136 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:49.254094    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:46:49.327509 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:49.327464    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lqgv\" (UniqueName: \"kubernetes.io/projected/55004416-09dd-42f4-a4e3-520c3e7a0094-kube-api-access-4lqgv\") pod \"console-57946b7674-fjvqx\" (UID: \"55004416-09dd-42f4-a4e3-520c3e7a0094\") " pod="openshift-console/console-57946b7674-fjvqx"
Sep 14 09:46:49.327509 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:49.327515    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/55004416-09dd-42f4-a4e3-520c3e7a0094-service-ca\") pod \"console-57946b7674-fjvqx\" (UID: \"55004416-09dd-42f4-a4e3-520c3e7a0094\") " pod="openshift-console/console-57946b7674-fjvqx"
Sep 14 09:46:49.327775 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:49.327546    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/55004416-09dd-42f4-a4e3-520c3e7a0094-oauth-serving-cert\") pod \"console-57946b7674-fjvqx\" (UID: \"55004416-09dd-42f4-a4e3-520c3e7a0094\") " pod="openshift-console/console-57946b7674-fjvqx"
Sep 14 09:46:49.327775 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:49.327584    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/55004416-09dd-42f4-a4e3-520c3e7a0094-console-serving-cert\") pod \"console-57946b7674-fjvqx\" (UID: \"55004416-09dd-42f4-a4e3-520c3e7a0094\") " pod="openshift-console/console-57946b7674-fjvqx"
Sep 14 09:46:49.327775 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:49.327647    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55004416-09dd-42f4-a4e3-520c3e7a0094-trusted-ca-bundle\") pod \"console-57946b7674-fjvqx\" (UID: \"55004416-09dd-42f4-a4e3-520c3e7a0094\") " pod="openshift-console/console-57946b7674-fjvqx"
Sep 14 09:46:49.327775 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:49.327717    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/55004416-09dd-42f4-a4e3-520c3e7a0094-console-oauth-config\") pod \"console-57946b7674-fjvqx\" (UID: \"55004416-09dd-42f4-a4e3-520c3e7a0094\") " pod="openshift-console/console-57946b7674-fjvqx"
Sep 14 09:46:49.327991 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:49.327777    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/55004416-09dd-42f4-a4e3-520c3e7a0094-console-config\") pod \"console-57946b7674-fjvqx\" (UID: \"55004416-09dd-42f4-a4e3-520c3e7a0094\") " pod="openshift-console/console-57946b7674-fjvqx"
Sep 14 09:46:49.328501 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:49.328469    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/55004416-09dd-42f4-a4e3-520c3e7a0094-oauth-serving-cert\") pod \"console-57946b7674-fjvqx\" (UID: \"55004416-09dd-42f4-a4e3-520c3e7a0094\") " pod="openshift-console/console-57946b7674-fjvqx"
Sep 14 09:46:49.328642 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:49.328527    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/55004416-09dd-42f4-a4e3-520c3e7a0094-service-ca\") pod \"console-57946b7674-fjvqx\" (UID: \"55004416-09dd-42f4-a4e3-520c3e7a0094\") " pod="openshift-console/console-57946b7674-fjvqx"
Sep 14 09:46:49.328642 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:49.328603    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/55004416-09dd-42f4-a4e3-520c3e7a0094-console-config\") pod \"console-57946b7674-fjvqx\" (UID: \"55004416-09dd-42f4-a4e3-520c3e7a0094\") " pod="openshift-console/console-57946b7674-fjvqx"
Sep 14 09:46:49.328758 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:49.328701    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55004416-09dd-42f4-a4e3-520c3e7a0094-trusted-ca-bundle\") pod \"console-57946b7674-fjvqx\" (UID: \"55004416-09dd-42f4-a4e3-520c3e7a0094\") " pod="openshift-console/console-57946b7674-fjvqx"
Sep 14 09:46:49.330380 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:49.330340    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/55004416-09dd-42f4-a4e3-520c3e7a0094-console-oauth-config\") pod \"console-57946b7674-fjvqx\" (UID: \"55004416-09dd-42f4-a4e3-520c3e7a0094\") " pod="openshift-console/console-57946b7674-fjvqx"
Sep 14 09:46:49.330481 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:49.330436    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/55004416-09dd-42f4-a4e3-520c3e7a0094-console-serving-cert\") pod \"console-57946b7674-fjvqx\" (UID: \"55004416-09dd-42f4-a4e3-520c3e7a0094\") " pod="openshift-console/console-57946b7674-fjvqx"
Sep 14 09:46:49.336109 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:49.336068    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lqgv\" (UniqueName: \"kubernetes.io/projected/55004416-09dd-42f4-a4e3-520c3e7a0094-kube-api-access-4lqgv\") pod \"console-57946b7674-fjvqx\" (UID: \"55004416-09dd-42f4-a4e3-520c3e7a0094\") " pod="openshift-console/console-57946b7674-fjvqx"
Sep 14 09:46:49.450831 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:49.450748    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57946b7674-fjvqx"
Sep 14 09:46:49.733791 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:49.733702    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57946b7674-fjvqx"]
Sep 14 09:46:49.737235 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:46:49.737204    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55004416_09dd_42f4_a4e3_520c3e7a0094.slice/crio-494e7f0c65a564d5275769b092b50afbab9460c4c09eace4a399ee5ad7ac326c WatchSource:0}: Error finding container 494e7f0c65a564d5275769b092b50afbab9460c4c09eace4a399ee5ad7ac326c: Status 404 returned error can't find the container with id 494e7f0c65a564d5275769b092b50afbab9460c4c09eace4a399ee5ad7ac326c
Sep 14 09:46:50.460477 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:50.460431    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57946b7674-fjvqx" event={"ID":"55004416-09dd-42f4-a4e3-520c3e7a0094","Type":"ContainerStarted","Data":"17ed7f8133de3ed988bbd11e33f70648c5bdc54f2b716663fa4350ef0ce85c10"}
Sep 14 09:46:50.460872 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:50.460534    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57946b7674-fjvqx" event={"ID":"55004416-09dd-42f4-a4e3-520c3e7a0094","Type":"ContainerStarted","Data":"494e7f0c65a564d5275769b092b50afbab9460c4c09eace4a399ee5ad7ac326c"}
Sep 14 09:46:50.478910 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:50.478837    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-57946b7674-fjvqx" podStartSLOduration=1.4788168769999999 podStartE2EDuration="1.478816877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:46:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:46:50.478542979 +0000 UTC m=+199.518217641" watchObservedRunningTime="2026-09-14 09:46:50.478816877 +0000 UTC m=+199.518491526"
Sep 14 09:46:52.729410 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:52.729374    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:46:59.451635 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:59.451592    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-57946b7674-fjvqx"
Sep 14 09:46:59.452073 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:59.451668    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-57946b7674-fjvqx"
Sep 14 09:46:59.456801 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:59.456769    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-57946b7674-fjvqx"
Sep 14 09:46:59.497932 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:59.497897    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-57946b7674-fjvqx"
Sep 14 09:46:59.698106 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:59.698065    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:46:59.794978 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:46:59.794938    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:47:28.625250 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:47:28.625191    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:47:36.396190 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:47:36.396156    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:47:36.460607 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:47:36.460562    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:47:57.550716 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:47:57.550618    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:47:57.645516 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:47:57.645476    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:47:58.311205 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:47:58.311169    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v6nqt"]
Sep 14 09:47:58.314541 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:47:58.314516    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v6nqt"
Sep 14 09:47:58.316824 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:47:58.316788    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-4r7sj\""
Sep 14 09:47:58.316974 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:47:58.316910    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 14 09:47:58.317359 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:47:58.317342    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 14 09:47:58.325285 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:47:58.325252    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v6nqt"]
Sep 14 09:47:58.373250 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:47:58.373176    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd5d6e39-7367-4b15-b7de-78f56dc1150c-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v6nqt\" (UID: \"dd5d6e39-7367-4b15-b7de-78f56dc1150c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v6nqt"
Sep 14 09:47:58.373250 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:47:58.373248    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd5d6e39-7367-4b15-b7de-78f56dc1150c-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v6nqt\" (UID: \"dd5d6e39-7367-4b15-b7de-78f56dc1150c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v6nqt"
Sep 14 09:47:58.373497 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:47:58.373317    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8qb4\" (UniqueName: \"kubernetes.io/projected/dd5d6e39-7367-4b15-b7de-78f56dc1150c-kube-api-access-b8qb4\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v6nqt\" (UID: \"dd5d6e39-7367-4b15-b7de-78f56dc1150c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v6nqt"
Sep 14 09:47:58.474551 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:47:58.474504    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8qb4\" (UniqueName: \"kubernetes.io/projected/dd5d6e39-7367-4b15-b7de-78f56dc1150c-kube-api-access-b8qb4\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v6nqt\" (UID: \"dd5d6e39-7367-4b15-b7de-78f56dc1150c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v6nqt"
Sep 14 09:47:58.474760 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:47:58.474663    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd5d6e39-7367-4b15-b7de-78f56dc1150c-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v6nqt\" (UID: \"dd5d6e39-7367-4b15-b7de-78f56dc1150c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v6nqt"
Sep 14 09:47:58.474760 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:47:58.474693    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd5d6e39-7367-4b15-b7de-78f56dc1150c-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v6nqt\" (UID: \"dd5d6e39-7367-4b15-b7de-78f56dc1150c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v6nqt"
Sep 14 09:47:58.475095 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:47:58.475076    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd5d6e39-7367-4b15-b7de-78f56dc1150c-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v6nqt\" (UID: \"dd5d6e39-7367-4b15-b7de-78f56dc1150c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v6nqt"
Sep 14 09:47:58.475095 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:47:58.475089    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd5d6e39-7367-4b15-b7de-78f56dc1150c-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v6nqt\" (UID: \"dd5d6e39-7367-4b15-b7de-78f56dc1150c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v6nqt"
Sep 14 09:47:58.483685 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:47:58.483653    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8qb4\" (UniqueName: \"kubernetes.io/projected/dd5d6e39-7367-4b15-b7de-78f56dc1150c-kube-api-access-b8qb4\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v6nqt\" (UID: \"dd5d6e39-7367-4b15-b7de-78f56dc1150c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v6nqt"
Sep 14 09:47:58.625028 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:47:58.624899    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v6nqt"
Sep 14 09:47:58.903347 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:47:58.903270    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v6nqt"]
Sep 14 09:47:58.906482 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:47:58.906452    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd5d6e39_7367_4b15_b7de_78f56dc1150c.slice/crio-8e6c5869bdbcd281f7a9199587f09ac056f622f0a4e89bd353b9e6f0f60be7b3 WatchSource:0}: Error finding container 8e6c5869bdbcd281f7a9199587f09ac056f622f0a4e89bd353b9e6f0f60be7b3: Status 404 returned error can't find the container with id 8e6c5869bdbcd281f7a9199587f09ac056f622f0a4e89bd353b9e6f0f60be7b3
Sep 14 09:47:59.722888 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:47:59.722847    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v6nqt" event={"ID":"dd5d6e39-7367-4b15-b7de-78f56dc1150c","Type":"ContainerStarted","Data":"8e6c5869bdbcd281f7a9199587f09ac056f622f0a4e89bd353b9e6f0f60be7b3"}
Sep 14 09:48:05.751062 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:05.751023    2563 generic.go:356] "Generic (PLEG): container finished" podID="dd5d6e39-7367-4b15-b7de-78f56dc1150c" containerID="19d8d934da8e21f5035131c0d6d61d94e45451793406ee22b6d0158cac0e9f28" exitCode=0
Sep 14 09:48:05.751425 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:05.751103    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v6nqt" event={"ID":"dd5d6e39-7367-4b15-b7de-78f56dc1150c","Type":"ContainerDied","Data":"19d8d934da8e21f5035131c0d6d61d94e45451793406ee22b6d0158cac0e9f28"}
Sep 14 09:48:23.823868 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:23.823834    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v6nqt" event={"ID":"dd5d6e39-7367-4b15-b7de-78f56dc1150c","Type":"ContainerStarted","Data":"18fef517d142276fe2a27fb1a79b3d7e5b524b101393581beff5a7dc59071b2f"}
Sep 14 09:48:24.830890 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:24.830846    2563 generic.go:356] "Generic (PLEG): container finished" podID="dd5d6e39-7367-4b15-b7de-78f56dc1150c" containerID="18fef517d142276fe2a27fb1a79b3d7e5b524b101393581beff5a7dc59071b2f" exitCode=0
Sep 14 09:48:24.831325 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:24.830930    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v6nqt" event={"ID":"dd5d6e39-7367-4b15-b7de-78f56dc1150c","Type":"ContainerDied","Data":"18fef517d142276fe2a27fb1a79b3d7e5b524b101393581beff5a7dc59071b2f"}
Sep 14 09:48:30.857915 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:30.857815    2563 generic.go:356] "Generic (PLEG): container finished" podID="dd5d6e39-7367-4b15-b7de-78f56dc1150c" containerID="349afaf787883f8becba65246b86f3c09b6a856430384fefdb1f007dd0cc521d" exitCode=0
Sep 14 09:48:30.858345 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:30.857906    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v6nqt" event={"ID":"dd5d6e39-7367-4b15-b7de-78f56dc1150c","Type":"ContainerDied","Data":"349afaf787883f8becba65246b86f3c09b6a856430384fefdb1f007dd0cc521d"}
Sep 14 09:48:31.362440 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:31.362411    2563 kubelet.go:1648] "Image garbage collection succeeded"
Sep 14 09:48:31.988936 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:31.988900    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v6nqt"
Sep 14 09:48:32.066530 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:32.066485    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd5d6e39-7367-4b15-b7de-78f56dc1150c-util\") pod \"dd5d6e39-7367-4b15-b7de-78f56dc1150c\" (UID: \"dd5d6e39-7367-4b15-b7de-78f56dc1150c\") "
Sep 14 09:48:32.066701 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:32.066587    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd5d6e39-7367-4b15-b7de-78f56dc1150c-bundle\") pod \"dd5d6e39-7367-4b15-b7de-78f56dc1150c\" (UID: \"dd5d6e39-7367-4b15-b7de-78f56dc1150c\") "
Sep 14 09:48:32.066701 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:32.066614    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b8qb4\" (UniqueName: \"kubernetes.io/projected/dd5d6e39-7367-4b15-b7de-78f56dc1150c-kube-api-access-b8qb4\") pod \"dd5d6e39-7367-4b15-b7de-78f56dc1150c\" (UID: \"dd5d6e39-7367-4b15-b7de-78f56dc1150c\") "
Sep 14 09:48:32.067298 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:32.067268    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd5d6e39-7367-4b15-b7de-78f56dc1150c-bundle" (OuterVolumeSpecName: "bundle") pod "dd5d6e39-7367-4b15-b7de-78f56dc1150c" (UID: "dd5d6e39-7367-4b15-b7de-78f56dc1150c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 09:48:32.068903 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:32.068874    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd5d6e39-7367-4b15-b7de-78f56dc1150c-kube-api-access-b8qb4" (OuterVolumeSpecName: "kube-api-access-b8qb4") pod "dd5d6e39-7367-4b15-b7de-78f56dc1150c" (UID: "dd5d6e39-7367-4b15-b7de-78f56dc1150c"). InnerVolumeSpecName "kube-api-access-b8qb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 09:48:32.072776 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:32.072734    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd5d6e39-7367-4b15-b7de-78f56dc1150c-util" (OuterVolumeSpecName: "util") pod "dd5d6e39-7367-4b15-b7de-78f56dc1150c" (UID: "dd5d6e39-7367-4b15-b7de-78f56dc1150c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 09:48:32.167651 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:32.167561    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd5d6e39-7367-4b15-b7de-78f56dc1150c-bundle\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:48:32.167651 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:32.167594    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8qb4\" (UniqueName: \"kubernetes.io/projected/dd5d6e39-7367-4b15-b7de-78f56dc1150c-kube-api-access-b8qb4\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:48:32.167651 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:32.167607    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd5d6e39-7367-4b15-b7de-78f56dc1150c-util\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:48:32.867692 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:32.867662    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v6nqt"
Sep 14 09:48:32.867840 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:32.867667    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v6nqt" event={"ID":"dd5d6e39-7367-4b15-b7de-78f56dc1150c","Type":"ContainerDied","Data":"8e6c5869bdbcd281f7a9199587f09ac056f622f0a4e89bd353b9e6f0f60be7b3"}
Sep 14 09:48:32.867840 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:32.867781    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e6c5869bdbcd281f7a9199587f09ac056f622f0a4e89bd353b9e6f0f60be7b3"
Sep 14 09:48:34.322141 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:34.322106    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:48:35.694359 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:35.694322    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:48:36.331065 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:36.331031    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-gddxr"]
Sep 14 09:48:36.331355 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:36.331339    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd5d6e39-7367-4b15-b7de-78f56dc1150c" containerName="util"
Sep 14 09:48:36.331355 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:36.331354    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd5d6e39-7367-4b15-b7de-78f56dc1150c" containerName="util"
Sep 14 09:48:36.331540 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:36.331375    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd5d6e39-7367-4b15-b7de-78f56dc1150c" containerName="pull"
Sep 14 09:48:36.331540 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:36.331381    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd5d6e39-7367-4b15-b7de-78f56dc1150c" containerName="pull"
Sep 14 09:48:36.331540 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:36.331394    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd5d6e39-7367-4b15-b7de-78f56dc1150c" containerName="extract"
Sep 14 09:48:36.331540 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:36.331399    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd5d6e39-7367-4b15-b7de-78f56dc1150c" containerName="extract"
Sep 14 09:48:36.331540 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:36.331467    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd5d6e39-7367-4b15-b7de-78f56dc1150c" containerName="extract"
Sep 14 09:48:36.334754 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:36.334727    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-gddxr"
Sep 14 09:48:36.336922 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:36.336897    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 14 09:48:36.337114 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:36.337098    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-7sdhx\""
Sep 14 09:48:36.337164 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:36.337124    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 14 09:48:36.351934 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:36.351885    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-gddxr"]
Sep 14 09:48:36.503287 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:36.503244    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d368179e-aeec-4427-9b52-4cffce67c5ff-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-gddxr\" (UID: \"d368179e-aeec-4427-9b52-4cffce67c5ff\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-gddxr"
Sep 14 09:48:36.503482 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:36.503331    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzntm\" (UniqueName: \"kubernetes.io/projected/d368179e-aeec-4427-9b52-4cffce67c5ff-kube-api-access-kzntm\") pod \"cert-manager-operator-controller-manager-55c444bf86-gddxr\" (UID: \"d368179e-aeec-4427-9b52-4cffce67c5ff\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-gddxr"
Sep 14 09:48:36.604161 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:36.604055    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d368179e-aeec-4427-9b52-4cffce67c5ff-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-gddxr\" (UID: \"d368179e-aeec-4427-9b52-4cffce67c5ff\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-gddxr"
Sep 14 09:48:36.604161 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:36.604126    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzntm\" (UniqueName: \"kubernetes.io/projected/d368179e-aeec-4427-9b52-4cffce67c5ff-kube-api-access-kzntm\") pod \"cert-manager-operator-controller-manager-55c444bf86-gddxr\" (UID: \"d368179e-aeec-4427-9b52-4cffce67c5ff\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-gddxr"
Sep 14 09:48:36.604528 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:36.604502    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d368179e-aeec-4427-9b52-4cffce67c5ff-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-gddxr\" (UID: \"d368179e-aeec-4427-9b52-4cffce67c5ff\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-gddxr"
Sep 14 09:48:36.613440 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:36.613404    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzntm\" (UniqueName: \"kubernetes.io/projected/d368179e-aeec-4427-9b52-4cffce67c5ff-kube-api-access-kzntm\") pod \"cert-manager-operator-controller-manager-55c444bf86-gddxr\" (UID: \"d368179e-aeec-4427-9b52-4cffce67c5ff\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-gddxr"
Sep 14 09:48:36.651175 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:36.651113    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-gddxr"
Sep 14 09:48:36.933350 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:36.933248    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-gddxr"]
Sep 14 09:48:36.939121 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:48:36.939063    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd368179e_aeec_4427_9b52_4cffce67c5ff.slice/crio-c7c9bb22a8b223a2d7d38601fc7d26ae5b8867731ceeca76ee304f916a8d3af1 WatchSource:0}: Error finding container c7c9bb22a8b223a2d7d38601fc7d26ae5b8867731ceeca76ee304f916a8d3af1: Status 404 returned error can't find the container with id c7c9bb22a8b223a2d7d38601fc7d26ae5b8867731ceeca76ee304f916a8d3af1
Sep 14 09:48:36.942185 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:36.942160    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 14 09:48:37.889391 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:37.889347    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-gddxr" event={"ID":"d368179e-aeec-4427-9b52-4cffce67c5ff","Type":"ContainerStarted","Data":"c7c9bb22a8b223a2d7d38601fc7d26ae5b8867731ceeca76ee304f916a8d3af1"}
Sep 14 09:48:37.908790 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:37.908747    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:48:38.009629 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:38.009593    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:48:38.896888 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:38.896792    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-gddxr" event={"ID":"d368179e-aeec-4427-9b52-4cffce67c5ff","Type":"ContainerStarted","Data":"8c4578b7023c7d83c955543530456c813de27fbd8e72ed4eb2dda20c0b4ff531"}
Sep 14 09:48:38.919013 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:38.918928    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-gddxr" podStartSLOduration=1.273311698 podStartE2EDuration="2.918906079s" totalImagesPullingTime="1.645594381s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:48:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:48:38.918575933 +0000 UTC m=+307.958250555" watchObservedRunningTime="2026-09-14 09:48:38.918906079 +0000 UTC m=+307.958580701"
Sep 14 09:48:39.407401 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:39.407360    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:48:39.509173 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:39.509130    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:48:40.220605 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:40.220565    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg7m97"]
Sep 14 09:48:40.224114 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:40.224086    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg7m97"
Sep 14 09:48:40.226300 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:40.226265    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-4r7sj\""
Sep 14 09:48:40.226300 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:40.226273    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 14 09:48:40.226650 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:40.226634    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 14 09:48:40.237619 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:40.237572    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg7m97"]
Sep 14 09:48:40.259985 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:40.259938    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:48:40.339314 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:40.339275    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znch7\" (UniqueName: \"kubernetes.io/projected/e6769cb9-0fdc-4fc1-a5e4-24041a0b0b5c-kube-api-access-znch7\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg7m97\" (UID: \"e6769cb9-0fdc-4fc1-a5e4-24041a0b0b5c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg7m97"
Sep 14 09:48:40.339498 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:40.339336    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e6769cb9-0fdc-4fc1-a5e4-24041a0b0b5c-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg7m97\" (UID: \"e6769cb9-0fdc-4fc1-a5e4-24041a0b0b5c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg7m97"
Sep 14 09:48:40.339498 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:40.339398    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e6769cb9-0fdc-4fc1-a5e4-24041a0b0b5c-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg7m97\" (UID: \"e6769cb9-0fdc-4fc1-a5e4-24041a0b0b5c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg7m97"
Sep 14 09:48:40.349378 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:40.349345    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:48:40.440937 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:40.440892    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e6769cb9-0fdc-4fc1-a5e4-24041a0b0b5c-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg7m97\" (UID: \"e6769cb9-0fdc-4fc1-a5e4-24041a0b0b5c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg7m97"
Sep 14 09:48:40.440937 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:40.440940    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e6769cb9-0fdc-4fc1-a5e4-24041a0b0b5c-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg7m97\" (UID: \"e6769cb9-0fdc-4fc1-a5e4-24041a0b0b5c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg7m97"
Sep 14 09:48:40.441489 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:40.441034    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znch7\" (UniqueName: \"kubernetes.io/projected/e6769cb9-0fdc-4fc1-a5e4-24041a0b0b5c-kube-api-access-znch7\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg7m97\" (UID: \"e6769cb9-0fdc-4fc1-a5e4-24041a0b0b5c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg7m97"
Sep 14 09:48:40.441489 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:40.441297    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e6769cb9-0fdc-4fc1-a5e4-24041a0b0b5c-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg7m97\" (UID: \"e6769cb9-0fdc-4fc1-a5e4-24041a0b0b5c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg7m97"
Sep 14 09:48:40.441489 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:40.441332    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e6769cb9-0fdc-4fc1-a5e4-24041a0b0b5c-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg7m97\" (UID: \"e6769cb9-0fdc-4fc1-a5e4-24041a0b0b5c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg7m97"
Sep 14 09:48:40.450764 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:40.450721    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znch7\" (UniqueName: \"kubernetes.io/projected/e6769cb9-0fdc-4fc1-a5e4-24041a0b0b5c-kube-api-access-znch7\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg7m97\" (UID: \"e6769cb9-0fdc-4fc1-a5e4-24041a0b0b5c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg7m97"
Sep 14 09:48:40.540713 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:40.540667    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg7m97"
Sep 14 09:48:40.821634 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:40.821531    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg7m97"]
Sep 14 09:48:40.823868 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:48:40.823837    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6769cb9_0fdc_4fc1_a5e4_24041a0b0b5c.slice/crio-147eec6bf3a69a7241fae2cc443e1da24797997df8a390e2218faf95a031161e WatchSource:0}: Error finding container 147eec6bf3a69a7241fae2cc443e1da24797997df8a390e2218faf95a031161e: Status 404 returned error can't find the container with id 147eec6bf3a69a7241fae2cc443e1da24797997df8a390e2218faf95a031161e
Sep 14 09:48:40.907921 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:40.907875    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg7m97" event={"ID":"e6769cb9-0fdc-4fc1-a5e4-24041a0b0b5c","Type":"ContainerStarted","Data":"4edde5e62cffb33fccce9b3141e98556c4c9654e423344cf819d3e95aa2f1cb2"}
Sep 14 09:48:40.908148 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:40.907970    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg7m97" event={"ID":"e6769cb9-0fdc-4fc1-a5e4-24041a0b0b5c","Type":"ContainerStarted","Data":"147eec6bf3a69a7241fae2cc443e1da24797997df8a390e2218faf95a031161e"}
Sep 14 09:48:41.175352 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:41.175262    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-zmq79"]
Sep 14 09:48:41.178495 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:41.178473    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-zmq79"
Sep 14 09:48:41.181116 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:41.181079    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 14 09:48:41.181281 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:41.181199    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 14 09:48:41.181590 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:41.181575    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-rb52s\""
Sep 14 09:48:41.192310 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:41.192273    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-zmq79"]
Sep 14 09:48:41.348520 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:41.348478    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjz7t\" (UniqueName: \"kubernetes.io/projected/2b2acf3d-b248-4856-a03b-3dd897616a90-kube-api-access-cjz7t\") pod \"cert-manager-webhook-6486ffd997-zmq79\" (UID: \"2b2acf3d-b248-4856-a03b-3dd897616a90\") " pod="cert-manager/cert-manager-webhook-6486ffd997-zmq79"
Sep 14 09:48:41.348706 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:41.348542    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2b2acf3d-b248-4856-a03b-3dd897616a90-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-zmq79\" (UID: \"2b2acf3d-b248-4856-a03b-3dd897616a90\") " pod="cert-manager/cert-manager-webhook-6486ffd997-zmq79"
Sep 14 09:48:41.449768 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:41.449674    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjz7t\" (UniqueName: \"kubernetes.io/projected/2b2acf3d-b248-4856-a03b-3dd897616a90-kube-api-access-cjz7t\") pod \"cert-manager-webhook-6486ffd997-zmq79\" (UID: \"2b2acf3d-b248-4856-a03b-3dd897616a90\") " pod="cert-manager/cert-manager-webhook-6486ffd997-zmq79"
Sep 14 09:48:41.449768 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:41.449738    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2b2acf3d-b248-4856-a03b-3dd897616a90-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-zmq79\" (UID: \"2b2acf3d-b248-4856-a03b-3dd897616a90\") " pod="cert-manager/cert-manager-webhook-6486ffd997-zmq79"
Sep 14 09:48:41.458125 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:41.458094    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2b2acf3d-b248-4856-a03b-3dd897616a90-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-zmq79\" (UID: \"2b2acf3d-b248-4856-a03b-3dd897616a90\") " pod="cert-manager/cert-manager-webhook-6486ffd997-zmq79"
Sep 14 09:48:41.458261 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:41.458100    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjz7t\" (UniqueName: \"kubernetes.io/projected/2b2acf3d-b248-4856-a03b-3dd897616a90-kube-api-access-cjz7t\") pod \"cert-manager-webhook-6486ffd997-zmq79\" (UID: \"2b2acf3d-b248-4856-a03b-3dd897616a90\") " pod="cert-manager/cert-manager-webhook-6486ffd997-zmq79"
Sep 14 09:48:41.487780 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:41.487746    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-zmq79"
Sep 14 09:48:41.764549 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:41.764512    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-zmq79"]
Sep 14 09:48:41.766990 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:48:41.766965    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2b2acf3d_b248_4856_a03b_3dd897616a90.slice/crio-3daf71a17cf57156e13e3cb4b47db7186d757bffae595257f47519ce6bd7dbd8 WatchSource:0}: Error finding container 3daf71a17cf57156e13e3cb4b47db7186d757bffae595257f47519ce6bd7dbd8: Status 404 returned error can't find the container with id 3daf71a17cf57156e13e3cb4b47db7186d757bffae595257f47519ce6bd7dbd8
Sep 14 09:48:41.913911 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:41.913870    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-zmq79" event={"ID":"2b2acf3d-b248-4856-a03b-3dd897616a90","Type":"ContainerStarted","Data":"3daf71a17cf57156e13e3cb4b47db7186d757bffae595257f47519ce6bd7dbd8"}
Sep 14 09:48:41.915488 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:41.915460    2563 generic.go:356] "Generic (PLEG): container finished" podID="e6769cb9-0fdc-4fc1-a5e4-24041a0b0b5c" containerID="4edde5e62cffb33fccce9b3141e98556c4c9654e423344cf819d3e95aa2f1cb2" exitCode=0
Sep 14 09:48:41.915614 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:41.915500    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg7m97" event={"ID":"e6769cb9-0fdc-4fc1-a5e4-24041a0b0b5c","Type":"ContainerDied","Data":"4edde5e62cffb33fccce9b3141e98556c4c9654e423344cf819d3e95aa2f1cb2"}
Sep 14 09:48:42.469759 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:42.469697    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:48:43.943860 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:43.943824    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:48:44.742022 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:44.741964    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:48:44.811204 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:44.811152    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:48:45.937491 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:45.937451    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-zmq79" event={"ID":"2b2acf3d-b248-4856-a03b-3dd897616a90","Type":"ContainerStarted","Data":"1cb06e76d77b82f82c64d01099f0c164ab7ff7640bb85b9e0b3a5aff759638a2"}
Sep 14 09:48:45.937955 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:45.937530    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-zmq79"
Sep 14 09:48:45.939538 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:45.939509    2563 generic.go:356] "Generic (PLEG): container finished" podID="e6769cb9-0fdc-4fc1-a5e4-24041a0b0b5c" containerID="72d46c401a8a8de5c0e592c59075ecade7f4041c114a8ecaee54a5653c4a8738" exitCode=0
Sep 14 09:48:45.939652 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:45.939590    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg7m97" event={"ID":"e6769cb9-0fdc-4fc1-a5e4-24041a0b0b5c","Type":"ContainerDied","Data":"72d46c401a8a8de5c0e592c59075ecade7f4041c114a8ecaee54a5653c4a8738"}
Sep 14 09:48:45.961179 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:45.961110    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-zmq79" podStartSLOduration=1.776199187 podStartE2EDuration="4.961086257s" totalImagesPullingTime="3.18488707s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:48:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:48:45.959014097 +0000 UTC m=+314.998688712" watchObservedRunningTime="2026-09-14 09:48:45.961086257 +0000 UTC m=+315.000760897"
Sep 14 09:48:46.945916 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:46.945879    2563 generic.go:356] "Generic (PLEG): container finished" podID="e6769cb9-0fdc-4fc1-a5e4-24041a0b0b5c" containerID="0e26137a69da68573679b71fc7ad6e318b7e1c912336ae47da0394ef854b4ab9" exitCode=0
Sep 14 09:48:46.946339 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:46.945965    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg7m97" event={"ID":"e6769cb9-0fdc-4fc1-a5e4-24041a0b0b5c","Type":"ContainerDied","Data":"0e26137a69da68573679b71fc7ad6e318b7e1c912336ae47da0394ef854b4ab9"}
Sep 14 09:48:47.340211 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:47.340158    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:48:47.429744 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:47.429704    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:48:48.087529 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:48.087495    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg7m97"
Sep 14 09:48:48.103011 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:48.102964    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-znch7\" (UniqueName: \"kubernetes.io/projected/e6769cb9-0fdc-4fc1-a5e4-24041a0b0b5c-kube-api-access-znch7\") pod \"e6769cb9-0fdc-4fc1-a5e4-24041a0b0b5c\" (UID: \"e6769cb9-0fdc-4fc1-a5e4-24041a0b0b5c\") "
Sep 14 09:48:48.103205 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:48.103035    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e6769cb9-0fdc-4fc1-a5e4-24041a0b0b5c-util\") pod \"e6769cb9-0fdc-4fc1-a5e4-24041a0b0b5c\" (UID: \"e6769cb9-0fdc-4fc1-a5e4-24041a0b0b5c\") "
Sep 14 09:48:48.103205 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:48.103086    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e6769cb9-0fdc-4fc1-a5e4-24041a0b0b5c-bundle\") pod \"e6769cb9-0fdc-4fc1-a5e4-24041a0b0b5c\" (UID: \"e6769cb9-0fdc-4fc1-a5e4-24041a0b0b5c\") "
Sep 14 09:48:48.103519 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:48.103485    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6769cb9-0fdc-4fc1-a5e4-24041a0b0b5c-bundle" (OuterVolumeSpecName: "bundle") pod "e6769cb9-0fdc-4fc1-a5e4-24041a0b0b5c" (UID: "e6769cb9-0fdc-4fc1-a5e4-24041a0b0b5c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 09:48:48.105226 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:48.105198    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6769cb9-0fdc-4fc1-a5e4-24041a0b0b5c-kube-api-access-znch7" (OuterVolumeSpecName: "kube-api-access-znch7") pod "e6769cb9-0fdc-4fc1-a5e4-24041a0b0b5c" (UID: "e6769cb9-0fdc-4fc1-a5e4-24041a0b0b5c"). InnerVolumeSpecName "kube-api-access-znch7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 09:48:48.109332 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:48.109289    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6769cb9-0fdc-4fc1-a5e4-24041a0b0b5c-util" (OuterVolumeSpecName: "util") pod "e6769cb9-0fdc-4fc1-a5e4-24041a0b0b5c" (UID: "e6769cb9-0fdc-4fc1-a5e4-24041a0b0b5c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 09:48:48.203855 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:48.203772    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e6769cb9-0fdc-4fc1-a5e4-24041a0b0b5c-bundle\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:48:48.203855 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:48.203802    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znch7\" (UniqueName: \"kubernetes.io/projected/e6769cb9-0fdc-4fc1-a5e4-24041a0b0b5c-kube-api-access-znch7\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:48:48.203855 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:48.203813    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e6769cb9-0fdc-4fc1-a5e4-24041a0b0b5c-util\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:48:48.956658 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:48.956617    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg7m97" event={"ID":"e6769cb9-0fdc-4fc1-a5e4-24041a0b0b5c","Type":"ContainerDied","Data":"147eec6bf3a69a7241fae2cc443e1da24797997df8a390e2218faf95a031161e"}
Sep 14 09:48:48.956829 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:48.956672    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg7m97"
Sep 14 09:48:48.956829 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:48.956680    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="147eec6bf3a69a7241fae2cc443e1da24797997df8a390e2218faf95a031161e"
Sep 14 09:48:50.761268 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:50.761224    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-8xmwg"]
Sep 14 09:48:50.761655 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:50.761526    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6769cb9-0fdc-4fc1-a5e4-24041a0b0b5c" containerName="util"
Sep 14 09:48:50.761655 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:50.761537    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6769cb9-0fdc-4fc1-a5e4-24041a0b0b5c" containerName="util"
Sep 14 09:48:50.761655 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:50.761555    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6769cb9-0fdc-4fc1-a5e4-24041a0b0b5c" containerName="pull"
Sep 14 09:48:50.761655 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:50.761561    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6769cb9-0fdc-4fc1-a5e4-24041a0b0b5c" containerName="pull"
Sep 14 09:48:50.761655 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:50.761576    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6769cb9-0fdc-4fc1-a5e4-24041a0b0b5c" containerName="extract"
Sep 14 09:48:50.761655 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:50.761582    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6769cb9-0fdc-4fc1-a5e4-24041a0b0b5c" containerName="extract"
Sep 14 09:48:50.761655 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:50.761627    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6769cb9-0fdc-4fc1-a5e4-24041a0b0b5c" containerName="extract"
Sep 14 09:48:50.767192 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:50.767166    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-8xmwg"
Sep 14 09:48:50.769385 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:50.769366    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-k4cj7\""
Sep 14 09:48:50.781528 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:50.781496    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-8xmwg"]
Sep 14 09:48:50.826613 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:50.826579    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5f48f550-a8a6-4a19-a8eb-e7639e41e099-bound-sa-token\") pod \"cert-manager-55b4d954d5-8xmwg\" (UID: \"5f48f550-a8a6-4a19-a8eb-e7639e41e099\") " pod="cert-manager/cert-manager-55b4d954d5-8xmwg"
Sep 14 09:48:50.826785 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:50.826645    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvztv\" (UniqueName: \"kubernetes.io/projected/5f48f550-a8a6-4a19-a8eb-e7639e41e099-kube-api-access-jvztv\") pod \"cert-manager-55b4d954d5-8xmwg\" (UID: \"5f48f550-a8a6-4a19-a8eb-e7639e41e099\") " pod="cert-manager/cert-manager-55b4d954d5-8xmwg"
Sep 14 09:48:50.831181 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:50.831147    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:48:50.927970 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:50.927915    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5f48f550-a8a6-4a19-a8eb-e7639e41e099-bound-sa-token\") pod \"cert-manager-55b4d954d5-8xmwg\" (UID: \"5f48f550-a8a6-4a19-a8eb-e7639e41e099\") " pod="cert-manager/cert-manager-55b4d954d5-8xmwg"
Sep 14 09:48:50.928159 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:50.928046    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvztv\" (UniqueName: \"kubernetes.io/projected/5f48f550-a8a6-4a19-a8eb-e7639e41e099-kube-api-access-jvztv\") pod \"cert-manager-55b4d954d5-8xmwg\" (UID: \"5f48f550-a8a6-4a19-a8eb-e7639e41e099\") " pod="cert-manager/cert-manager-55b4d954d5-8xmwg"
Sep 14 09:48:50.936399 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:50.936356    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5f48f550-a8a6-4a19-a8eb-e7639e41e099-bound-sa-token\") pod \"cert-manager-55b4d954d5-8xmwg\" (UID: \"5f48f550-a8a6-4a19-a8eb-e7639e41e099\") " pod="cert-manager/cert-manager-55b4d954d5-8xmwg"
Sep 14 09:48:50.936583 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:50.936405    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvztv\" (UniqueName: \"kubernetes.io/projected/5f48f550-a8a6-4a19-a8eb-e7639e41e099-kube-api-access-jvztv\") pod \"cert-manager-55b4d954d5-8xmwg\" (UID: \"5f48f550-a8a6-4a19-a8eb-e7639e41e099\") " pod="cert-manager/cert-manager-55b4d954d5-8xmwg"
Sep 14 09:48:51.076497 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:51.076399    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-8xmwg"
Sep 14 09:48:51.351353 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:51.351237    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-8xmwg"]
Sep 14 09:48:51.353634 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:48:51.353601    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f48f550_a8a6_4a19_a8eb_e7639e41e099.slice/crio-10c418c288adb4ca81bc9c58dbb91d8a229e632183ae77e77be89d772a7a684d WatchSource:0}: Error finding container 10c418c288adb4ca81bc9c58dbb91d8a229e632183ae77e77be89d772a7a684d: Status 404 returned error can't find the container with id 10c418c288adb4ca81bc9c58dbb91d8a229e632183ae77e77be89d772a7a684d
Sep 14 09:48:51.949057 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:51.949019    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-zmq79"
Sep 14 09:48:51.971499 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:51.971458    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-8xmwg" event={"ID":"5f48f550-a8a6-4a19-a8eb-e7639e41e099","Type":"ContainerStarted","Data":"bbb214bd7a75ad8afa94b9e8be234db2bd7050ddf6e6810e3bff831bbc4a443a"}
Sep 14 09:48:51.971499 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:51.971505    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-8xmwg" event={"ID":"5f48f550-a8a6-4a19-a8eb-e7639e41e099","Type":"ContainerStarted","Data":"10c418c288adb4ca81bc9c58dbb91d8a229e632183ae77e77be89d772a7a684d"}
Sep 14 09:48:51.993839 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:51.993775    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-8xmwg" podStartSLOduration=1.993754942 podStartE2EDuration="1.993754942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:48:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:48:51.993384721 +0000 UTC m=+321.033059343" watchObservedRunningTime="2026-09-14 09:48:51.993754942 +0000 UTC m=+321.033429564"
Sep 14 09:48:55.061666 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:55.061629    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:48:55.151400 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:55.151367    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:48:56.125569 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:56.125533    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:48:56.225566 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:48:56.225532    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:49:02.011050 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:02.010974    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:49:02.114899 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:02.114850    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:49:03.583506 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:03.583466    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7jd8k"]
Sep 14 09:49:03.588692 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:03.588659    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7jd8k"
Sep 14 09:49:03.590871 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:03.590841    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-4r7sj\""
Sep 14 09:49:03.591079 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:03.590943    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 14 09:49:03.591319 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:03.591301    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 14 09:49:03.597152 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:03.596977    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7jd8k"]
Sep 14 09:49:03.647252 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:03.647216    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/229dbccd-14a6-4c99-9c41-36d71d9e7661-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7jd8k\" (UID: \"229dbccd-14a6-4c99-9c41-36d71d9e7661\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7jd8k"
Sep 14 09:49:03.647441 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:03.647270    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d49db\" (UniqueName: \"kubernetes.io/projected/229dbccd-14a6-4c99-9c41-36d71d9e7661-kube-api-access-d49db\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7jd8k\" (UID: \"229dbccd-14a6-4c99-9c41-36d71d9e7661\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7jd8k"
Sep 14 09:49:03.647441 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:03.647300    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/229dbccd-14a6-4c99-9c41-36d71d9e7661-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7jd8k\" (UID: \"229dbccd-14a6-4c99-9c41-36d71d9e7661\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7jd8k"
Sep 14 09:49:03.748461 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:03.748407    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d49db\" (UniqueName: \"kubernetes.io/projected/229dbccd-14a6-4c99-9c41-36d71d9e7661-kube-api-access-d49db\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7jd8k\" (UID: \"229dbccd-14a6-4c99-9c41-36d71d9e7661\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7jd8k"
Sep 14 09:49:03.748461 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:03.748468    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/229dbccd-14a6-4c99-9c41-36d71d9e7661-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7jd8k\" (UID: \"229dbccd-14a6-4c99-9c41-36d71d9e7661\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7jd8k"
Sep 14 09:49:03.748689 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:03.748534    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/229dbccd-14a6-4c99-9c41-36d71d9e7661-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7jd8k\" (UID: \"229dbccd-14a6-4c99-9c41-36d71d9e7661\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7jd8k"
Sep 14 09:49:03.748987 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:03.748964    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/229dbccd-14a6-4c99-9c41-36d71d9e7661-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7jd8k\" (UID: \"229dbccd-14a6-4c99-9c41-36d71d9e7661\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7jd8k"
Sep 14 09:49:03.749056 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:03.748980    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/229dbccd-14a6-4c99-9c41-36d71d9e7661-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7jd8k\" (UID: \"229dbccd-14a6-4c99-9c41-36d71d9e7661\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7jd8k"
Sep 14 09:49:03.757073 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:03.757035    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d49db\" (UniqueName: \"kubernetes.io/projected/229dbccd-14a6-4c99-9c41-36d71d9e7661-kube-api-access-d49db\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7jd8k\" (UID: \"229dbccd-14a6-4c99-9c41-36d71d9e7661\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7jd8k"
Sep 14 09:49:03.899556 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:03.899459    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7jd8k"
Sep 14 09:49:04.178552 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:04.178523    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7jd8k"]
Sep 14 09:49:04.181334 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:49:04.181298    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod229dbccd_14a6_4c99_9c41_36d71d9e7661.slice/crio-46eb6a8dc5fdd4920a3711df83462cbfceb6d19530d73ac5e3307aab2c22e03d WatchSource:0}: Error finding container 46eb6a8dc5fdd4920a3711df83462cbfceb6d19530d73ac5e3307aab2c22e03d: Status 404 returned error can't find the container with id 46eb6a8dc5fdd4920a3711df83462cbfceb6d19530d73ac5e3307aab2c22e03d
Sep 14 09:49:04.193385 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:04.193348    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:49:05.038509 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:05.038419    2563 generic.go:356] "Generic (PLEG): container finished" podID="229dbccd-14a6-4c99-9c41-36d71d9e7661" containerID="37292dac4ecdec272a0ef5ad95261d01fe284849a53d38951ba82cf3c3fc4a3e" exitCode=0
Sep 14 09:49:05.038874 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:05.038507    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7jd8k" event={"ID":"229dbccd-14a6-4c99-9c41-36d71d9e7661","Type":"ContainerDied","Data":"37292dac4ecdec272a0ef5ad95261d01fe284849a53d38951ba82cf3c3fc4a3e"}
Sep 14 09:49:05.038874 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:05.038560    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7jd8k" event={"ID":"229dbccd-14a6-4c99-9c41-36d71d9e7661","Type":"ContainerStarted","Data":"46eb6a8dc5fdd4920a3711df83462cbfceb6d19530d73ac5e3307aab2c22e03d"}
Sep 14 09:49:06.044790 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:06.044753    2563 generic.go:356] "Generic (PLEG): container finished" podID="229dbccd-14a6-4c99-9c41-36d71d9e7661" containerID="f2f6b00e8123975bad31d626d4048d2005738a57e280c1feee71a9d193535b69" exitCode=0
Sep 14 09:49:06.045311 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:06.044805    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7jd8k" event={"ID":"229dbccd-14a6-4c99-9c41-36d71d9e7661","Type":"ContainerDied","Data":"f2f6b00e8123975bad31d626d4048d2005738a57e280c1feee71a9d193535b69"}
Sep 14 09:49:07.051802 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:07.051766    2563 generic.go:356] "Generic (PLEG): container finished" podID="229dbccd-14a6-4c99-9c41-36d71d9e7661" containerID="c92ea9b951cbc79e1d4f618283d207601fc2e24c4dd37c447368e9a10a918f98" exitCode=0
Sep 14 09:49:07.052242 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:07.051855    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7jd8k" event={"ID":"229dbccd-14a6-4c99-9c41-36d71d9e7661","Type":"ContainerDied","Data":"c92ea9b951cbc79e1d4f618283d207601fc2e24c4dd37c447368e9a10a918f98"}
Sep 14 09:49:08.189097 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:08.189063    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7jd8k"
Sep 14 09:49:08.285529 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:08.285484    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/229dbccd-14a6-4c99-9c41-36d71d9e7661-bundle\") pod \"229dbccd-14a6-4c99-9c41-36d71d9e7661\" (UID: \"229dbccd-14a6-4c99-9c41-36d71d9e7661\") "
Sep 14 09:49:08.285730 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:08.285546    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/229dbccd-14a6-4c99-9c41-36d71d9e7661-util\") pod \"229dbccd-14a6-4c99-9c41-36d71d9e7661\" (UID: \"229dbccd-14a6-4c99-9c41-36d71d9e7661\") "
Sep 14 09:49:08.285730 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:08.285586    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d49db\" (UniqueName: \"kubernetes.io/projected/229dbccd-14a6-4c99-9c41-36d71d9e7661-kube-api-access-d49db\") pod \"229dbccd-14a6-4c99-9c41-36d71d9e7661\" (UID: \"229dbccd-14a6-4c99-9c41-36d71d9e7661\") "
Sep 14 09:49:08.286299 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:08.286268    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/229dbccd-14a6-4c99-9c41-36d71d9e7661-bundle" (OuterVolumeSpecName: "bundle") pod "229dbccd-14a6-4c99-9c41-36d71d9e7661" (UID: "229dbccd-14a6-4c99-9c41-36d71d9e7661"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 09:49:08.287733 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:08.287705    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/229dbccd-14a6-4c99-9c41-36d71d9e7661-kube-api-access-d49db" (OuterVolumeSpecName: "kube-api-access-d49db") pod "229dbccd-14a6-4c99-9c41-36d71d9e7661" (UID: "229dbccd-14a6-4c99-9c41-36d71d9e7661"). InnerVolumeSpecName "kube-api-access-d49db". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 09:49:08.291241 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:08.291203    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/229dbccd-14a6-4c99-9c41-36d71d9e7661-util" (OuterVolumeSpecName: "util") pod "229dbccd-14a6-4c99-9c41-36d71d9e7661" (UID: "229dbccd-14a6-4c99-9c41-36d71d9e7661"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 09:49:08.387154 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:08.387057    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/229dbccd-14a6-4c99-9c41-36d71d9e7661-bundle\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:49:08.387154 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:08.387089    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/229dbccd-14a6-4c99-9c41-36d71d9e7661-util\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:49:08.387154 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:08.387100    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d49db\" (UniqueName: \"kubernetes.io/projected/229dbccd-14a6-4c99-9c41-36d71d9e7661-kube-api-access-d49db\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:49:09.064426 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:09.064395    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7jd8k"
Sep 14 09:49:09.064597 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:09.064394    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7jd8k" event={"ID":"229dbccd-14a6-4c99-9c41-36d71d9e7661","Type":"ContainerDied","Data":"46eb6a8dc5fdd4920a3711df83462cbfceb6d19530d73ac5e3307aab2c22e03d"}
Sep 14 09:49:09.064597 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:09.064515    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46eb6a8dc5fdd4920a3711df83462cbfceb6d19530d73ac5e3307aab2c22e03d"
Sep 14 09:49:20.354110 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:20.354072    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02xh2b"]
Sep 14 09:49:20.354497 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:20.354395    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="229dbccd-14a6-4c99-9c41-36d71d9e7661" containerName="util"
Sep 14 09:49:20.354497 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:20.354407    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="229dbccd-14a6-4c99-9c41-36d71d9e7661" containerName="util"
Sep 14 09:49:20.354497 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:20.354417    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="229dbccd-14a6-4c99-9c41-36d71d9e7661" containerName="pull"
Sep 14 09:49:20.354497 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:20.354423    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="229dbccd-14a6-4c99-9c41-36d71d9e7661" containerName="pull"
Sep 14 09:49:20.354497 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:20.354432    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="229dbccd-14a6-4c99-9c41-36d71d9e7661" containerName="extract"
Sep 14 09:49:20.354497 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:20.354437    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="229dbccd-14a6-4c99-9c41-36d71d9e7661" containerName="extract"
Sep 14 09:49:20.354497 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:20.354494    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="229dbccd-14a6-4c99-9c41-36d71d9e7661" containerName="extract"
Sep 14 09:49:20.357019 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:20.356973    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02xh2b"
Sep 14 09:49:20.359177 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:20.359150    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 14 09:49:20.359281 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:20.359150    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 14 09:49:20.359654 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:20.359633    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-4r7sj\""
Sep 14 09:49:20.367747 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:20.367715    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02xh2b"]
Sep 14 09:49:20.495915 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:20.495879    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3f5f8e52-a24c-4915-ba2c-f603abb6f83d-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02xh2b\" (UID: \"3f5f8e52-a24c-4915-ba2c-f603abb6f83d\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02xh2b"
Sep 14 09:49:20.496125 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:20.495927    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3f5f8e52-a24c-4915-ba2c-f603abb6f83d-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02xh2b\" (UID: \"3f5f8e52-a24c-4915-ba2c-f603abb6f83d\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02xh2b"
Sep 14 09:49:20.496125 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:20.495969    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6mjz\" (UniqueName: \"kubernetes.io/projected/3f5f8e52-a24c-4915-ba2c-f603abb6f83d-kube-api-access-g6mjz\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02xh2b\" (UID: \"3f5f8e52-a24c-4915-ba2c-f603abb6f83d\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02xh2b"
Sep 14 09:49:20.544793 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:20.544760    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:49:20.596843 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:20.596800    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3f5f8e52-a24c-4915-ba2c-f603abb6f83d-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02xh2b\" (UID: \"3f5f8e52-a24c-4915-ba2c-f603abb6f83d\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02xh2b"
Sep 14 09:49:20.597090 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:20.596857    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6mjz\" (UniqueName: \"kubernetes.io/projected/3f5f8e52-a24c-4915-ba2c-f603abb6f83d-kube-api-access-g6mjz\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02xh2b\" (UID: \"3f5f8e52-a24c-4915-ba2c-f603abb6f83d\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02xh2b"
Sep 14 09:49:20.597090 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:20.596918    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3f5f8e52-a24c-4915-ba2c-f603abb6f83d-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02xh2b\" (UID: \"3f5f8e52-a24c-4915-ba2c-f603abb6f83d\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02xh2b"
Sep 14 09:49:20.597306 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:20.597282    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3f5f8e52-a24c-4915-ba2c-f603abb6f83d-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02xh2b\" (UID: \"3f5f8e52-a24c-4915-ba2c-f603abb6f83d\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02xh2b"
Sep 14 09:49:20.597353 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:20.597293    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3f5f8e52-a24c-4915-ba2c-f603abb6f83d-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02xh2b\" (UID: \"3f5f8e52-a24c-4915-ba2c-f603abb6f83d\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02xh2b"
Sep 14 09:49:20.625728 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:20.625649    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6mjz\" (UniqueName: \"kubernetes.io/projected/3f5f8e52-a24c-4915-ba2c-f603abb6f83d-kube-api-access-g6mjz\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02xh2b\" (UID: \"3f5f8e52-a24c-4915-ba2c-f603abb6f83d\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02xh2b"
Sep 14 09:49:20.663306 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:20.663270    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:49:20.667175 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:20.667145    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02xh2b"
Sep 14 09:49:20.944468 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:20.944440    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02xh2b"]
Sep 14 09:49:20.947144 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:49:20.947111    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f5f8e52_a24c_4915_ba2c_f603abb6f83d.slice/crio-2c85b026c6a4f23e899e070cffe0120408529b185bab6bb4c4a5224e3af5e8e8 WatchSource:0}: Error finding container 2c85b026c6a4f23e899e070cffe0120408529b185bab6bb4c4a5224e3af5e8e8: Status 404 returned error can't find the container with id 2c85b026c6a4f23e899e070cffe0120408529b185bab6bb4c4a5224e3af5e8e8
Sep 14 09:49:21.120848 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:21.120814    2563 generic.go:356] "Generic (PLEG): container finished" podID="3f5f8e52-a24c-4915-ba2c-f603abb6f83d" containerID="62a842927f37bb21d9addc3894f1a50b4a86de470d570a0ed31e834e7ffe540e" exitCode=0
Sep 14 09:49:21.121081 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:21.120907    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02xh2b" event={"ID":"3f5f8e52-a24c-4915-ba2c-f603abb6f83d","Type":"ContainerDied","Data":"62a842927f37bb21d9addc3894f1a50b4a86de470d570a0ed31e834e7ffe540e"}
Sep 14 09:49:21.121081 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:21.120966    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02xh2b" event={"ID":"3f5f8e52-a24c-4915-ba2c-f603abb6f83d","Type":"ContainerStarted","Data":"2c85b026c6a4f23e899e070cffe0120408529b185bab6bb4c4a5224e3af5e8e8"}
Sep 14 09:49:21.146430 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:21.146387    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:49:21.401431 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:21.401395    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:49:21.506069 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:21.505616    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:49:21.549957 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:21.549920    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-84fdd9869c-4v5sx"]
Sep 14 09:49:21.552676 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:21.552632    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-84fdd9869c-4v5sx"
Sep 14 09:49:21.555241 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:21.555088    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 14 09:49:21.555241 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:21.555133    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-n4vsj\""
Sep 14 09:49:21.555494 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:21.555324    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 14 09:49:21.555494 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:21.555322    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 14 09:49:21.555494 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:21.555415    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 14 09:49:21.575579 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:21.575543    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-84fdd9869c-4v5sx"]
Sep 14 09:49:21.708323 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:21.708238    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a452960c-7230-4b45-9b8b-999b0123033d-tmp\") pod \"opendatahub-operator-controller-manager-84fdd9869c-4v5sx\" (UID: \"a452960c-7230-4b45-9b8b-999b0123033d\") " pod="opendatahub/opendatahub-operator-controller-manager-84fdd9869c-4v5sx"
Sep 14 09:49:21.708323 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:21.708315    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a452960c-7230-4b45-9b8b-999b0123033d-webhook-cert\") pod \"opendatahub-operator-controller-manager-84fdd9869c-4v5sx\" (UID: \"a452960c-7230-4b45-9b8b-999b0123033d\") " pod="opendatahub/opendatahub-operator-controller-manager-84fdd9869c-4v5sx"
Sep 14 09:49:21.708557 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:21.708412    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a452960c-7230-4b45-9b8b-999b0123033d-manifests\") pod \"opendatahub-operator-controller-manager-84fdd9869c-4v5sx\" (UID: \"a452960c-7230-4b45-9b8b-999b0123033d\") " pod="opendatahub/opendatahub-operator-controller-manager-84fdd9869c-4v5sx"
Sep 14 09:49:21.708557 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:21.708462    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a452960c-7230-4b45-9b8b-999b0123033d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84fdd9869c-4v5sx\" (UID: \"a452960c-7230-4b45-9b8b-999b0123033d\") " pod="opendatahub/opendatahub-operator-controller-manager-84fdd9869c-4v5sx"
Sep 14 09:49:21.708557 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:21.708539    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqc5t\" (UniqueName: \"kubernetes.io/projected/a452960c-7230-4b45-9b8b-999b0123033d-kube-api-access-jqc5t\") pod \"opendatahub-operator-controller-manager-84fdd9869c-4v5sx\" (UID: \"a452960c-7230-4b45-9b8b-999b0123033d\") " pod="opendatahub/opendatahub-operator-controller-manager-84fdd9869c-4v5sx"
Sep 14 09:49:21.794781 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:21.794749    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:49:21.809182 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:21.809146    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a452960c-7230-4b45-9b8b-999b0123033d-manifests\") pod \"opendatahub-operator-controller-manager-84fdd9869c-4v5sx\" (UID: \"a452960c-7230-4b45-9b8b-999b0123033d\") " pod="opendatahub/opendatahub-operator-controller-manager-84fdd9869c-4v5sx"
Sep 14 09:49:21.809182 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:21.809186    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a452960c-7230-4b45-9b8b-999b0123033d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84fdd9869c-4v5sx\" (UID: \"a452960c-7230-4b45-9b8b-999b0123033d\") " pod="opendatahub/opendatahub-operator-controller-manager-84fdd9869c-4v5sx"
Sep 14 09:49:21.809415 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:21.809257    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqc5t\" (UniqueName: \"kubernetes.io/projected/a452960c-7230-4b45-9b8b-999b0123033d-kube-api-access-jqc5t\") pod \"opendatahub-operator-controller-manager-84fdd9869c-4v5sx\" (UID: \"a452960c-7230-4b45-9b8b-999b0123033d\") " pod="opendatahub/opendatahub-operator-controller-manager-84fdd9869c-4v5sx"
Sep 14 09:49:21.809415 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:21.809290    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a452960c-7230-4b45-9b8b-999b0123033d-tmp\") pod \"opendatahub-operator-controller-manager-84fdd9869c-4v5sx\" (UID: \"a452960c-7230-4b45-9b8b-999b0123033d\") " pod="opendatahub/opendatahub-operator-controller-manager-84fdd9869c-4v5sx"
Sep 14 09:49:21.809415 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:21.809333    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a452960c-7230-4b45-9b8b-999b0123033d-webhook-cert\") pod \"opendatahub-operator-controller-manager-84fdd9869c-4v5sx\" (UID: \"a452960c-7230-4b45-9b8b-999b0123033d\") " pod="opendatahub/opendatahub-operator-controller-manager-84fdd9869c-4v5sx"
Sep 14 09:49:21.809635 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:21.809608    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a452960c-7230-4b45-9b8b-999b0123033d-manifests\") pod \"opendatahub-operator-controller-manager-84fdd9869c-4v5sx\" (UID: \"a452960c-7230-4b45-9b8b-999b0123033d\") " pod="opendatahub/opendatahub-operator-controller-manager-84fdd9869c-4v5sx"
Sep 14 09:49:21.811685 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:21.811655    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a452960c-7230-4b45-9b8b-999b0123033d-tmp\") pod \"opendatahub-operator-controller-manager-84fdd9869c-4v5sx\" (UID: \"a452960c-7230-4b45-9b8b-999b0123033d\") " pod="opendatahub/opendatahub-operator-controller-manager-84fdd9869c-4v5sx"
Sep 14 09:49:21.811798 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:21.811743    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a452960c-7230-4b45-9b8b-999b0123033d-webhook-cert\") pod \"opendatahub-operator-controller-manager-84fdd9869c-4v5sx\" (UID: \"a452960c-7230-4b45-9b8b-999b0123033d\") " pod="opendatahub/opendatahub-operator-controller-manager-84fdd9869c-4v5sx"
Sep 14 09:49:21.811798 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:21.811791    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a452960c-7230-4b45-9b8b-999b0123033d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84fdd9869c-4v5sx\" (UID: \"a452960c-7230-4b45-9b8b-999b0123033d\") " pod="opendatahub/opendatahub-operator-controller-manager-84fdd9869c-4v5sx"
Sep 14 09:49:21.824757 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:21.824719    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqc5t\" (UniqueName: \"kubernetes.io/projected/a452960c-7230-4b45-9b8b-999b0123033d-kube-api-access-jqc5t\") pod \"opendatahub-operator-controller-manager-84fdd9869c-4v5sx\" (UID: \"a452960c-7230-4b45-9b8b-999b0123033d\") " pod="opendatahub/opendatahub-operator-controller-manager-84fdd9869c-4v5sx"
Sep 14 09:49:21.865805 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:21.865765    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-84fdd9869c-4v5sx"
Sep 14 09:49:21.894571 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:21.894529    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:49:22.000345 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:22.000306    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:49:22.099356 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:22.099321    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:49:22.127311 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:22.127283    2563 generic.go:356] "Generic (PLEG): container finished" podID="3f5f8e52-a24c-4915-ba2c-f603abb6f83d" containerID="8de5b31ad51e24cc49bb908832f1dd7ce4c7b6af50b9b5a073fe788e3f09e6e0" exitCode=0
Sep 14 09:49:22.127475 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:22.127363    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02xh2b" event={"ID":"3f5f8e52-a24c-4915-ba2c-f603abb6f83d","Type":"ContainerDied","Data":"8de5b31ad51e24cc49bb908832f1dd7ce4c7b6af50b9b5a073fe788e3f09e6e0"}
Sep 14 09:49:22.154374 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:22.154298    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-84fdd9869c-4v5sx"]
Sep 14 09:49:22.158244 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:49:22.158188    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda452960c_7230_4b45_9b8b_999b0123033d.slice/crio-9ddbb242198b80d26d2484252cbaa55b93f3b1307f333bb95bef98f1cb54cc31 WatchSource:0}: Error finding container 9ddbb242198b80d26d2484252cbaa55b93f3b1307f333bb95bef98f1cb54cc31: Status 404 returned error can't find the container with id 9ddbb242198b80d26d2484252cbaa55b93f3b1307f333bb95bef98f1cb54cc31
Sep 14 09:49:22.197776 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:22.197730    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:49:23.136322 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:23.136281    2563 generic.go:356] "Generic (PLEG): container finished" podID="3f5f8e52-a24c-4915-ba2c-f603abb6f83d" containerID="a54431fbb1bb44b544ded4fe210ee0e72a4753a73903e0a3711438bdb63b4ce0" exitCode=0
Sep 14 09:49:23.136773 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:23.136388    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02xh2b" event={"ID":"3f5f8e52-a24c-4915-ba2c-f603abb6f83d","Type":"ContainerDied","Data":"a54431fbb1bb44b544ded4fe210ee0e72a4753a73903e0a3711438bdb63b4ce0"}
Sep 14 09:49:23.138509 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:23.138468    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84fdd9869c-4v5sx" event={"ID":"a452960c-7230-4b45-9b8b-999b0123033d","Type":"ContainerStarted","Data":"9ddbb242198b80d26d2484252cbaa55b93f3b1307f333bb95bef98f1cb54cc31"}
Sep 14 09:49:24.449973 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:24.449873    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:49:24.587847 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:24.587804    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:49:25.113211 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:25.113182    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02xh2b"
Sep 14 09:49:25.152070 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:25.152024    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02xh2b" event={"ID":"3f5f8e52-a24c-4915-ba2c-f603abb6f83d","Type":"ContainerDied","Data":"2c85b026c6a4f23e899e070cffe0120408529b185bab6bb4c4a5224e3af5e8e8"}
Sep 14 09:49:25.152256 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:25.152077    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02xh2b"
Sep 14 09:49:25.152256 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:25.152096    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c85b026c6a4f23e899e070cffe0120408529b185bab6bb4c4a5224e3af5e8e8"
Sep 14 09:49:25.240883 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:25.240839    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3f5f8e52-a24c-4915-ba2c-f603abb6f83d-bundle\") pod \"3f5f8e52-a24c-4915-ba2c-f603abb6f83d\" (UID: \"3f5f8e52-a24c-4915-ba2c-f603abb6f83d\") "
Sep 14 09:49:25.240883 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:25.240886    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g6mjz\" (UniqueName: \"kubernetes.io/projected/3f5f8e52-a24c-4915-ba2c-f603abb6f83d-kube-api-access-g6mjz\") pod \"3f5f8e52-a24c-4915-ba2c-f603abb6f83d\" (UID: \"3f5f8e52-a24c-4915-ba2c-f603abb6f83d\") "
Sep 14 09:49:25.241122 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:25.240903    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3f5f8e52-a24c-4915-ba2c-f603abb6f83d-util\") pod \"3f5f8e52-a24c-4915-ba2c-f603abb6f83d\" (UID: \"3f5f8e52-a24c-4915-ba2c-f603abb6f83d\") "
Sep 14 09:49:25.241612 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:25.241582    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f5f8e52-a24c-4915-ba2c-f603abb6f83d-bundle" (OuterVolumeSpecName: "bundle") pod "3f5f8e52-a24c-4915-ba2c-f603abb6f83d" (UID: "3f5f8e52-a24c-4915-ba2c-f603abb6f83d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 09:49:25.243110 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:25.243075    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f5f8e52-a24c-4915-ba2c-f603abb6f83d-kube-api-access-g6mjz" (OuterVolumeSpecName: "kube-api-access-g6mjz") pod "3f5f8e52-a24c-4915-ba2c-f603abb6f83d" (UID: "3f5f8e52-a24c-4915-ba2c-f603abb6f83d"). InnerVolumeSpecName "kube-api-access-g6mjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 09:49:25.246640 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:25.246607    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f5f8e52-a24c-4915-ba2c-f603abb6f83d-util" (OuterVolumeSpecName: "util") pod "3f5f8e52-a24c-4915-ba2c-f603abb6f83d" (UID: "3f5f8e52-a24c-4915-ba2c-f603abb6f83d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 09:49:25.341475 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:25.341434    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3f5f8e52-a24c-4915-ba2c-f603abb6f83d-bundle\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:49:25.341475 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:25.341467    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6mjz\" (UniqueName: \"kubernetes.io/projected/3f5f8e52-a24c-4915-ba2c-f603abb6f83d-kube-api-access-g6mjz\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:49:25.341475 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:25.341479    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3f5f8e52-a24c-4915-ba2c-f603abb6f83d-util\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:49:26.158358 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:26.158323    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84fdd9869c-4v5sx" event={"ID":"a452960c-7230-4b45-9b8b-999b0123033d","Type":"ContainerStarted","Data":"211d96fd8e0da2705c494a09fcaf2960a9626ebc67e8992b0187b579b8c68b41"}
Sep 14 09:49:27.164489 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:27.164447    2563 generic.go:356] "Generic (PLEG): container finished" podID="a452960c-7230-4b45-9b8b-999b0123033d" containerID="211d96fd8e0da2705c494a09fcaf2960a9626ebc67e8992b0187b579b8c68b41" exitCode=0
Sep 14 09:49:27.164892 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:27.164536    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84fdd9869c-4v5sx" event={"ID":"a452960c-7230-4b45-9b8b-999b0123033d","Type":"ContainerDied","Data":"211d96fd8e0da2705c494a09fcaf2960a9626ebc67e8992b0187b579b8c68b41"}
Sep 14 09:49:28.172417 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:28.172374    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84fdd9869c-4v5sx" event={"ID":"a452960c-7230-4b45-9b8b-999b0123033d","Type":"ContainerStarted","Data":"87130cc8e1299659505dc8295bfdf1cdd74749b7ce7681b2fd60848b8f9a3430"}
Sep 14 09:49:28.172830 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:28.172536    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-84fdd9869c-4v5sx"
Sep 14 09:49:28.198035 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:28.197959    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-84fdd9869c-4v5sx" podStartSLOduration=2.646856258 podStartE2EDuration="7.197937132s" totalImagesPullingTime="3.522350985s" totalInitContainerRuntime="1.028729889s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:49:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:49:28.196476632 +0000 UTC m=+357.236151255" watchObservedRunningTime="2026-09-14 09:49:28.197937132 +0000 UTC m=+357.237611772"
Sep 14 09:49:32.832170 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:32.832133    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:49:32.933548 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:32.933505    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:49:33.298393 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:33.298357    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:49:33.396123 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:33.396087    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:49:39.179703 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:39.179670    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-84fdd9869c-4v5sx"
Sep 14 09:49:41.077797 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:41.077760    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:49:41.195815 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:41.195768    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:49:41.558432 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:41.558391    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:49:41.629416 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:41.629377    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-65d9bcd789-c4mnb"]
Sep 14 09:49:41.629718 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:41.629703    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f5f8e52-a24c-4915-ba2c-f603abb6f83d" containerName="extract"
Sep 14 09:49:41.629763 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:41.629722    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f5f8e52-a24c-4915-ba2c-f603abb6f83d" containerName="extract"
Sep 14 09:49:41.629763 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:41.629738    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f5f8e52-a24c-4915-ba2c-f603abb6f83d" containerName="pull"
Sep 14 09:49:41.629763 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:41.629744    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f5f8e52-a24c-4915-ba2c-f603abb6f83d" containerName="pull"
Sep 14 09:49:41.629763 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:41.629752    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f5f8e52-a24c-4915-ba2c-f603abb6f83d" containerName="util"
Sep 14 09:49:41.629763 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:41.629757    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f5f8e52-a24c-4915-ba2c-f603abb6f83d" containerName="util"
Sep 14 09:49:41.629917 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:41.629810    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f5f8e52-a24c-4915-ba2c-f603abb6f83d" containerName="extract"
Sep 14 09:49:41.641065 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:41.641027    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-65d9bcd789-c4mnb"
Sep 14 09:49:41.643826 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:41.643796    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 14 09:49:41.644079 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:41.643910    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 14 09:49:41.644079 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:41.644029    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-x6wrj\""
Sep 14 09:49:41.644239 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:41.644167    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 14 09:49:41.644357 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:41.644331    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 14 09:49:41.645777 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:41.645751    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-65d9bcd789-c4mnb"]
Sep 14 09:49:41.669816 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:41.669785    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:49:41.674099 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:41.674066    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dea6846-7ff5-4959-9d8c-2c7b04b85f60-tls-certs\") pod \"kube-auth-proxy-65d9bcd789-c4mnb\" (UID: \"9dea6846-7ff5-4959-9d8c-2c7b04b85f60\") " pod="openshift-ingress/kube-auth-proxy-65d9bcd789-c4mnb"
Sep 14 09:49:41.674221 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:41.674120    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpttd\" (UniqueName: \"kubernetes.io/projected/9dea6846-7ff5-4959-9d8c-2c7b04b85f60-kube-api-access-wpttd\") pod \"kube-auth-proxy-65d9bcd789-c4mnb\" (UID: \"9dea6846-7ff5-4959-9d8c-2c7b04b85f60\") " pod="openshift-ingress/kube-auth-proxy-65d9bcd789-c4mnb"
Sep 14 09:49:41.674221 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:41.674201    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9dea6846-7ff5-4959-9d8c-2c7b04b85f60-tmp\") pod \"kube-auth-proxy-65d9bcd789-c4mnb\" (UID: \"9dea6846-7ff5-4959-9d8c-2c7b04b85f60\") " pod="openshift-ingress/kube-auth-proxy-65d9bcd789-c4mnb"
Sep 14 09:49:41.774572 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:41.774526    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9dea6846-7ff5-4959-9d8c-2c7b04b85f60-tmp\") pod \"kube-auth-proxy-65d9bcd789-c4mnb\" (UID: \"9dea6846-7ff5-4959-9d8c-2c7b04b85f60\") " pod="openshift-ingress/kube-auth-proxy-65d9bcd789-c4mnb"
Sep 14 09:49:41.774777 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:41.774589    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dea6846-7ff5-4959-9d8c-2c7b04b85f60-tls-certs\") pod \"kube-auth-proxy-65d9bcd789-c4mnb\" (UID: \"9dea6846-7ff5-4959-9d8c-2c7b04b85f60\") " pod="openshift-ingress/kube-auth-proxy-65d9bcd789-c4mnb"
Sep 14 09:49:41.774777 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:41.774622    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpttd\" (UniqueName: \"kubernetes.io/projected/9dea6846-7ff5-4959-9d8c-2c7b04b85f60-kube-api-access-wpttd\") pod \"kube-auth-proxy-65d9bcd789-c4mnb\" (UID: \"9dea6846-7ff5-4959-9d8c-2c7b04b85f60\") " pod="openshift-ingress/kube-auth-proxy-65d9bcd789-c4mnb"
Sep 14 09:49:41.776987 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:41.776958    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9dea6846-7ff5-4959-9d8c-2c7b04b85f60-tmp\") pod \"kube-auth-proxy-65d9bcd789-c4mnb\" (UID: \"9dea6846-7ff5-4959-9d8c-2c7b04b85f60\") " pod="openshift-ingress/kube-auth-proxy-65d9bcd789-c4mnb"
Sep 14 09:49:41.777141 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:41.777120    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dea6846-7ff5-4959-9d8c-2c7b04b85f60-tls-certs\") pod \"kube-auth-proxy-65d9bcd789-c4mnb\" (UID: \"9dea6846-7ff5-4959-9d8c-2c7b04b85f60\") " pod="openshift-ingress/kube-auth-proxy-65d9bcd789-c4mnb"
Sep 14 09:49:41.782861 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:41.782829    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpttd\" (UniqueName: \"kubernetes.io/projected/9dea6846-7ff5-4959-9d8c-2c7b04b85f60-kube-api-access-wpttd\") pod \"kube-auth-proxy-65d9bcd789-c4mnb\" (UID: \"9dea6846-7ff5-4959-9d8c-2c7b04b85f60\") " pod="openshift-ingress/kube-auth-proxy-65d9bcd789-c4mnb"
Sep 14 09:49:41.952973 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:41.952877    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-65d9bcd789-c4mnb"
Sep 14 09:49:42.240271 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:42.240247    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-65d9bcd789-c4mnb"]
Sep 14 09:49:42.242868 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:49:42.242837    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dea6846_7ff5_4959_9d8c_2c7b04b85f60.slice/crio-d8b0189dbdcfb67c8dc79d889603e365f61cdd47a8513b95785f5ce7fb4f2581 WatchSource:0}: Error finding container d8b0189dbdcfb67c8dc79d889603e365f61cdd47a8513b95785f5ce7fb4f2581: Status 404 returned error can't find the container with id d8b0189dbdcfb67c8dc79d889603e365f61cdd47a8513b95785f5ce7fb4f2581
Sep 14 09:49:43.219723 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:43.219670    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:49:43.247762 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:43.247702    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-65d9bcd789-c4mnb" event={"ID":"9dea6846-7ff5-4959-9d8c-2c7b04b85f60","Type":"ContainerStarted","Data":"d8b0189dbdcfb67c8dc79d889603e365f61cdd47a8513b95785f5ce7fb4f2581"}
Sep 14 09:49:43.349539 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:43.349475    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:49:44.519143 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:44.519102    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:49:45.258673 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:45.258623    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-65d9bcd789-c4mnb" event={"ID":"9dea6846-7ff5-4959-9d8c-2c7b04b85f60","Type":"ContainerStarted","Data":"1b59e066b8f9b789af0f2cd1cb73f4521a02bbe47c1f65b5cfeb78638f6c2b90"}
Sep 14 09:49:45.273365 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:45.273296    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-65d9bcd789-c4mnb" podStartSLOduration=1.572460896 podStartE2EDuration="4.273272873s" totalImagesPullingTime="2.700811977s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:49:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:49:45.27327102 +0000 UTC m=+374.312945654" watchObservedRunningTime="2026-09-14 09:49:45.273272873 +0000 UTC m=+374.312947495"
Sep 14 09:49:48.481017 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:48.480970    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:49:48.581464 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:48.581427    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:49:49.135833 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:49.135794    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lvspj"]
Sep 14 09:49:49.139328 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:49.139303    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lvspj"
Sep 14 09:49:49.141504 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:49.141479    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 14 09:49:49.141658 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:49.141479    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-4q84w\""
Sep 14 09:49:49.150726 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:49.150688    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lvspj"]
Sep 14 09:49:49.169043 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:49.168978    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:49:49.244732 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:49.241435    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1436e28-a6d7-42eb-a096-f0970a2145b4-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-lvspj\" (UID: \"e1436e28-a6d7-42eb-a096-f0970a2145b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lvspj"
Sep 14 09:49:49.244732 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:49.241522    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgv46\" (UniqueName: \"kubernetes.io/projected/e1436e28-a6d7-42eb-a096-f0970a2145b4-kube-api-access-jgv46\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-lvspj\" (UID: \"e1436e28-a6d7-42eb-a096-f0970a2145b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lvspj"
Sep 14 09:49:49.244732 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:49.241564    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1436e28-a6d7-42eb-a096-f0970a2145b4-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-lvspj\" (UID: \"e1436e28-a6d7-42eb-a096-f0970a2145b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lvspj"
Sep 14 09:49:49.244732 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:49.241603    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1436e28-a6d7-42eb-a096-f0970a2145b4-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-lvspj\" (UID: \"e1436e28-a6d7-42eb-a096-f0970a2145b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lvspj"
Sep 14 09:49:49.244732 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:49.241639    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1436e28-a6d7-42eb-a096-f0970a2145b4-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-lvspj\" (UID: \"e1436e28-a6d7-42eb-a096-f0970a2145b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lvspj"
Sep 14 09:49:49.244732 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:49.241681    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1436e28-a6d7-42eb-a096-f0970a2145b4-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-lvspj\" (UID: \"e1436e28-a6d7-42eb-a096-f0970a2145b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lvspj"
Sep 14 09:49:49.244732 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:49.241724    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1436e28-a6d7-42eb-a096-f0970a2145b4-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-lvspj\" (UID: \"e1436e28-a6d7-42eb-a096-f0970a2145b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lvspj"
Sep 14 09:49:49.244732 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:49.241762    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1436e28-a6d7-42eb-a096-f0970a2145b4-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-lvspj\" (UID: \"e1436e28-a6d7-42eb-a096-f0970a2145b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lvspj"
Sep 14 09:49:49.244732 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:49.241804    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1436e28-a6d7-42eb-a096-f0970a2145b4-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-lvspj\" (UID: \"e1436e28-a6d7-42eb-a096-f0970a2145b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lvspj"
Sep 14 09:49:49.342480 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:49.342434    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1436e28-a6d7-42eb-a096-f0970a2145b4-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-lvspj\" (UID: \"e1436e28-a6d7-42eb-a096-f0970a2145b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lvspj"
Sep 14 09:49:49.342480 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:49.342488    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1436e28-a6d7-42eb-a096-f0970a2145b4-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-lvspj\" (UID: \"e1436e28-a6d7-42eb-a096-f0970a2145b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lvspj"
Sep 14 09:49:49.342732 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:49.342521    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgv46\" (UniqueName: \"kubernetes.io/projected/e1436e28-a6d7-42eb-a096-f0970a2145b4-kube-api-access-jgv46\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-lvspj\" (UID: \"e1436e28-a6d7-42eb-a096-f0970a2145b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lvspj"
Sep 14 09:49:49.342732 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:49.342544    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1436e28-a6d7-42eb-a096-f0970a2145b4-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-lvspj\" (UID: \"e1436e28-a6d7-42eb-a096-f0970a2145b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lvspj"
Sep 14 09:49:49.342732 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:49.342562    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1436e28-a6d7-42eb-a096-f0970a2145b4-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-lvspj\" (UID: \"e1436e28-a6d7-42eb-a096-f0970a2145b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lvspj"
Sep 14 09:49:49.342732 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:49.342582    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1436e28-a6d7-42eb-a096-f0970a2145b4-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-lvspj\" (UID: \"e1436e28-a6d7-42eb-a096-f0970a2145b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lvspj"
Sep 14 09:49:49.342732 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:49.342609    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1436e28-a6d7-42eb-a096-f0970a2145b4-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-lvspj\" (UID: \"e1436e28-a6d7-42eb-a096-f0970a2145b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lvspj"
Sep 14 09:49:49.342952 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:49.342644    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1436e28-a6d7-42eb-a096-f0970a2145b4-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-lvspj\" (UID: \"e1436e28-a6d7-42eb-a096-f0970a2145b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lvspj"
Sep 14 09:49:49.343033 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:49.342946    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1436e28-a6d7-42eb-a096-f0970a2145b4-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-lvspj\" (UID: \"e1436e28-a6d7-42eb-a096-f0970a2145b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lvspj"
Sep 14 09:49:49.343164 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:49.343139    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1436e28-a6d7-42eb-a096-f0970a2145b4-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-lvspj\" (UID: \"e1436e28-a6d7-42eb-a096-f0970a2145b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lvspj"
Sep 14 09:49:49.343232 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:49.343212    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1436e28-a6d7-42eb-a096-f0970a2145b4-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-lvspj\" (UID: \"e1436e28-a6d7-42eb-a096-f0970a2145b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lvspj"
Sep 14 09:49:49.343301 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:49.343276    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1436e28-a6d7-42eb-a096-f0970a2145b4-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-lvspj\" (UID: \"e1436e28-a6d7-42eb-a096-f0970a2145b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lvspj"
Sep 14 09:49:49.343357 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:49.343334    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1436e28-a6d7-42eb-a096-f0970a2145b4-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-lvspj\" (UID: \"e1436e28-a6d7-42eb-a096-f0970a2145b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lvspj"
Sep 14 09:49:49.343584 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:49.343562    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1436e28-a6d7-42eb-a096-f0970a2145b4-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-lvspj\" (UID: \"e1436e28-a6d7-42eb-a096-f0970a2145b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lvspj"
Sep 14 09:49:49.345165 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:49.345134    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1436e28-a6d7-42eb-a096-f0970a2145b4-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-lvspj\" (UID: \"e1436e28-a6d7-42eb-a096-f0970a2145b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lvspj"
Sep 14 09:49:49.345418 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:49.345394    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1436e28-a6d7-42eb-a096-f0970a2145b4-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-lvspj\" (UID: \"e1436e28-a6d7-42eb-a096-f0970a2145b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lvspj"
Sep 14 09:49:49.353903 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:49.353867    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgv46\" (UniqueName: \"kubernetes.io/projected/e1436e28-a6d7-42eb-a096-f0970a2145b4-kube-api-access-jgv46\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-lvspj\" (UID: \"e1436e28-a6d7-42eb-a096-f0970a2145b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lvspj"
Sep 14 09:49:49.354265 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:49.354238    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1436e28-a6d7-42eb-a096-f0970a2145b4-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-lvspj\" (UID: \"e1436e28-a6d7-42eb-a096-f0970a2145b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lvspj"
Sep 14 09:49:49.372026 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:49.371960    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:49:49.451473 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:49.451374    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lvspj"
Sep 14 09:49:49.748152 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:49.748115    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lvspj"]
Sep 14 09:49:49.861968 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:49.861926    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:49:50.283547 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:50.283507    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lvspj" event={"ID":"e1436e28-a6d7-42eb-a096-f0970a2145b4","Type":"ContainerStarted","Data":"d329483e3c72b352b299ca4929beffc6919f75b7425984d1b71faf4a7a5fc2af"}
Sep 14 09:49:52.208934 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:52.208887    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:49:52.282916 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:52.282874    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:49:52.385756 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:52.385711    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:49:52.553701 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:52.553661    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:49:52.702808 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:52.702765    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-rlcpv"]
Sep 14 09:49:52.712147 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:52.712115    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-rlcpv"
Sep 14 09:49:52.717753 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:52.717716    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 14 09:49:52.722698 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:52.722408    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-rlcpv"]
Sep 14 09:49:52.731658 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:52.731612    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:49:52.776116 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:52.776074    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b419264c-1e5e-407a-9505-38c1b8a22721-tls-certs\") pod \"model-serving-api-54d89bd5fc-rlcpv\" (UID: \"b419264c-1e5e-407a-9505-38c1b8a22721\") " pod="opendatahub/model-serving-api-54d89bd5fc-rlcpv"
Sep 14 09:49:52.776319 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:52.776204    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6mhm\" (UniqueName: \"kubernetes.io/projected/b419264c-1e5e-407a-9505-38c1b8a22721-kube-api-access-w6mhm\") pod \"model-serving-api-54d89bd5fc-rlcpv\" (UID: \"b419264c-1e5e-407a-9505-38c1b8a22721\") " pod="opendatahub/model-serving-api-54d89bd5fc-rlcpv"
Sep 14 09:49:52.877538 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:52.877446    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6mhm\" (UniqueName: \"kubernetes.io/projected/b419264c-1e5e-407a-9505-38c1b8a22721-kube-api-access-w6mhm\") pod \"model-serving-api-54d89bd5fc-rlcpv\" (UID: \"b419264c-1e5e-407a-9505-38c1b8a22721\") " pod="opendatahub/model-serving-api-54d89bd5fc-rlcpv"
Sep 14 09:49:52.877538 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:52.877538    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b419264c-1e5e-407a-9505-38c1b8a22721-tls-certs\") pod \"model-serving-api-54d89bd5fc-rlcpv\" (UID: \"b419264c-1e5e-407a-9505-38c1b8a22721\") " pod="opendatahub/model-serving-api-54d89bd5fc-rlcpv"
Sep 14 09:49:52.877719 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:49:52.877641    2563 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 14 09:49:52.877719 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:49:52.877694    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b419264c-1e5e-407a-9505-38c1b8a22721-tls-certs podName:b419264c-1e5e-407a-9505-38c1b8a22721 nodeName:}" failed. No retries permitted until 2026-09-14 09:49:53.377677282 +0000 UTC m=+382.417351881 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/b419264c-1e5e-407a-9505-38c1b8a22721-tls-certs") pod "model-serving-api-54d89bd5fc-rlcpv" (UID: "b419264c-1e5e-407a-9505-38c1b8a22721") : secret "model-serving-api-tls" not found
Sep 14 09:49:52.886292 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:52.886255    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6mhm\" (UniqueName: \"kubernetes.io/projected/b419264c-1e5e-407a-9505-38c1b8a22721-kube-api-access-w6mhm\") pod \"model-serving-api-54d89bd5fc-rlcpv\" (UID: \"b419264c-1e5e-407a-9505-38c1b8a22721\") " pod="opendatahub/model-serving-api-54d89bd5fc-rlcpv"
Sep 14 09:49:52.899864 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:52.899823    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 09:49:52.900019 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:52.899918    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 09:49:52.900097 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:52.900032    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 09:49:53.305633 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:53.305587    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lvspj" event={"ID":"e1436e28-a6d7-42eb-a096-f0970a2145b4","Type":"ContainerStarted","Data":"8f099219d82f252d17d4d0396765fbe17bee2d1bc0702c56d55b9069dc8bfc9a"}
Sep 14 09:49:53.328801 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:53.328739    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lvspj" podStartSLOduration=1.177624755 podStartE2EDuration="4.328718683s" totalImagesPullingTime="3.151093928s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:49:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:49:53.326945355 +0000 UTC m=+382.366619978" watchObservedRunningTime="2026-09-14 09:49:53.328718683 +0000 UTC m=+382.368393360"
Sep 14 09:49:53.382698 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:53.382652    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b419264c-1e5e-407a-9505-38c1b8a22721-tls-certs\") pod \"model-serving-api-54d89bd5fc-rlcpv\" (UID: \"b419264c-1e5e-407a-9505-38c1b8a22721\") " pod="opendatahub/model-serving-api-54d89bd5fc-rlcpv"
Sep 14 09:49:53.382898 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:49:53.382749    2563 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 14 09:49:53.382898 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:49:53.382799    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b419264c-1e5e-407a-9505-38c1b8a22721-tls-certs podName:b419264c-1e5e-407a-9505-38c1b8a22721 nodeName:}" failed. No retries permitted until 2026-09-14 09:49:54.382785527 +0000 UTC m=+383.422460126 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/b419264c-1e5e-407a-9505-38c1b8a22721-tls-certs") pod "model-serving-api-54d89bd5fc-rlcpv" (UID: "b419264c-1e5e-407a-9505-38c1b8a22721") : secret "model-serving-api-tls" not found
Sep 14 09:49:53.452273 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:53.452229    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lvspj"
Sep 14 09:49:53.457296 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:53.457265    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lvspj"
Sep 14 09:49:53.472327 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:53.472291    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:49:54.310405 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:54.310368    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lvspj"
Sep 14 09:49:54.311425 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:54.311405    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-lvspj"
Sep 14 09:49:54.392832 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:54.392792    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b419264c-1e5e-407a-9505-38c1b8a22721-tls-certs\") pod \"model-serving-api-54d89bd5fc-rlcpv\" (UID: \"b419264c-1e5e-407a-9505-38c1b8a22721\") " pod="opendatahub/model-serving-api-54d89bd5fc-rlcpv"
Sep 14 09:49:54.395496 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:54.395465    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b419264c-1e5e-407a-9505-38c1b8a22721-tls-certs\") pod \"model-serving-api-54d89bd5fc-rlcpv\" (UID: \"b419264c-1e5e-407a-9505-38c1b8a22721\") " pod="opendatahub/model-serving-api-54d89bd5fc-rlcpv"
Sep 14 09:49:54.527973 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:54.527913    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-rlcpv"
Sep 14 09:49:54.817277 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:54.817111    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-rlcpv"]
Sep 14 09:49:54.819513 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:49:54.819479    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb419264c_1e5e_407a_9505_38c1b8a22721.slice/crio-18c899882fc09b99778b4796ecef04f734998acb517fee5aa2821f5fa331960d WatchSource:0}: Error finding container 18c899882fc09b99778b4796ecef04f734998acb517fee5aa2821f5fa331960d: Status 404 returned error can't find the container with id 18c899882fc09b99778b4796ecef04f734998acb517fee5aa2821f5fa331960d
Sep 14 09:49:55.316428 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:55.316388    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-rlcpv" event={"ID":"b419264c-1e5e-407a-9505-38c1b8a22721","Type":"ContainerStarted","Data":"18c899882fc09b99778b4796ecef04f734998acb517fee5aa2821f5fa331960d"}
Sep 14 09:49:57.328133 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:57.328092    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-rlcpv" event={"ID":"b419264c-1e5e-407a-9505-38c1b8a22721","Type":"ContainerStarted","Data":"e78dc8468e8bc54441fc43bd450438404c8b176835404d7feefd58f7dcf6923b"}
Sep 14 09:49:57.328589 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:57.328176    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-rlcpv"
Sep 14 09:49:57.347418 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:57.347358    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-rlcpv" podStartSLOduration=3.783129648 podStartE2EDuration="5.347340246s" totalImagesPullingTime="1.564210598s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:49:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:49:57.345974471 +0000 UTC m=+386.385649106" watchObservedRunningTime="2026-09-14 09:49:57.347340246 +0000 UTC m=+386.387014868"
Sep 14 09:49:58.396521 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:58.396485    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:49:58.589629 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:58.589582    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:49:58.691784 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:58.691662    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:49:59.673864 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:59.673817    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5d6ddf947f-kmx7m"]
Sep 14 09:49:59.677220 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:59.677196    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5d6ddf947f-kmx7m"
Sep 14 09:49:59.679711 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:59.679677    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 14 09:49:59.680481 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:59.680453    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 14 09:49:59.680623 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:59.680509    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 14 09:49:59.680623 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:59.680461    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-q9hzq\""
Sep 14 09:49:59.680623 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:59.680461    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 14 09:49:59.680623 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:59.680558    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 14 09:49:59.688920 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:59.688889    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5d6ddf947f-kmx7m"]
Sep 14 09:49:59.746754 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:59.746714    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0521996d-31be-49c0-84ca-a3ba08d80d71-metrics-cert\") pod \"lws-controller-manager-5d6ddf947f-kmx7m\" (UID: \"0521996d-31be-49c0-84ca-a3ba08d80d71\") " pod="openshift-lws-operator/lws-controller-manager-5d6ddf947f-kmx7m"
Sep 14 09:49:59.746754 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:59.746759    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cxtx\" (UniqueName: \"kubernetes.io/projected/0521996d-31be-49c0-84ca-a3ba08d80d71-kube-api-access-7cxtx\") pod \"lws-controller-manager-5d6ddf947f-kmx7m\" (UID: \"0521996d-31be-49c0-84ca-a3ba08d80d71\") " pod="openshift-lws-operator/lws-controller-manager-5d6ddf947f-kmx7m"
Sep 14 09:49:59.746988 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:59.746823    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0521996d-31be-49c0-84ca-a3ba08d80d71-cert\") pod \"lws-controller-manager-5d6ddf947f-kmx7m\" (UID: \"0521996d-31be-49c0-84ca-a3ba08d80d71\") " pod="openshift-lws-operator/lws-controller-manager-5d6ddf947f-kmx7m"
Sep 14 09:49:59.746988 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:59.746898    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0521996d-31be-49c0-84ca-a3ba08d80d71-manager-config\") pod \"lws-controller-manager-5d6ddf947f-kmx7m\" (UID: \"0521996d-31be-49c0-84ca-a3ba08d80d71\") " pod="openshift-lws-operator/lws-controller-manager-5d6ddf947f-kmx7m"
Sep 14 09:49:59.847950 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:59.847894    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0521996d-31be-49c0-84ca-a3ba08d80d71-manager-config\") pod \"lws-controller-manager-5d6ddf947f-kmx7m\" (UID: \"0521996d-31be-49c0-84ca-a3ba08d80d71\") " pod="openshift-lws-operator/lws-controller-manager-5d6ddf947f-kmx7m"
Sep 14 09:49:59.848161 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:59.847985    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0521996d-31be-49c0-84ca-a3ba08d80d71-metrics-cert\") pod \"lws-controller-manager-5d6ddf947f-kmx7m\" (UID: \"0521996d-31be-49c0-84ca-a3ba08d80d71\") " pod="openshift-lws-operator/lws-controller-manager-5d6ddf947f-kmx7m"
Sep 14 09:49:59.848161 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:59.848043    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cxtx\" (UniqueName: \"kubernetes.io/projected/0521996d-31be-49c0-84ca-a3ba08d80d71-kube-api-access-7cxtx\") pod \"lws-controller-manager-5d6ddf947f-kmx7m\" (UID: \"0521996d-31be-49c0-84ca-a3ba08d80d71\") " pod="openshift-lws-operator/lws-controller-manager-5d6ddf947f-kmx7m"
Sep 14 09:49:59.848161 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:59.848075    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0521996d-31be-49c0-84ca-a3ba08d80d71-cert\") pod \"lws-controller-manager-5d6ddf947f-kmx7m\" (UID: \"0521996d-31be-49c0-84ca-a3ba08d80d71\") " pod="openshift-lws-operator/lws-controller-manager-5d6ddf947f-kmx7m"
Sep 14 09:49:59.848643 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:59.848619    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0521996d-31be-49c0-84ca-a3ba08d80d71-manager-config\") pod \"lws-controller-manager-5d6ddf947f-kmx7m\" (UID: \"0521996d-31be-49c0-84ca-a3ba08d80d71\") " pod="openshift-lws-operator/lws-controller-manager-5d6ddf947f-kmx7m"
Sep 14 09:49:59.850711 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:59.850684    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0521996d-31be-49c0-84ca-a3ba08d80d71-cert\") pod \"lws-controller-manager-5d6ddf947f-kmx7m\" (UID: \"0521996d-31be-49c0-84ca-a3ba08d80d71\") " pod="openshift-lws-operator/lws-controller-manager-5d6ddf947f-kmx7m"
Sep 14 09:49:59.850814 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:59.850745    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0521996d-31be-49c0-84ca-a3ba08d80d71-metrics-cert\") pod \"lws-controller-manager-5d6ddf947f-kmx7m\" (UID: \"0521996d-31be-49c0-84ca-a3ba08d80d71\") " pod="openshift-lws-operator/lws-controller-manager-5d6ddf947f-kmx7m"
Sep 14 09:49:59.859139 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:59.859106    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cxtx\" (UniqueName: \"kubernetes.io/projected/0521996d-31be-49c0-84ca-a3ba08d80d71-kube-api-access-7cxtx\") pod \"lws-controller-manager-5d6ddf947f-kmx7m\" (UID: \"0521996d-31be-49c0-84ca-a3ba08d80d71\") " pod="openshift-lws-operator/lws-controller-manager-5d6ddf947f-kmx7m"
Sep 14 09:49:59.988523 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:49:59.988488    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5d6ddf947f-kmx7m"
Sep 14 09:50:00.266437 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:00.266408    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5d6ddf947f-kmx7m"]
Sep 14 09:50:00.268617 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:50:00.268587    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0521996d_31be_49c0_84ca_a3ba08d80d71.slice/crio-e6949d2692fe8c2f2be924d9ae9eacf01ab8eb9fcccaaab52c4b1342a62faa03 WatchSource:0}: Error finding container e6949d2692fe8c2f2be924d9ae9eacf01ab8eb9fcccaaab52c4b1342a62faa03: Status 404 returned error can't find the container with id e6949d2692fe8c2f2be924d9ae9eacf01ab8eb9fcccaaab52c4b1342a62faa03
Sep 14 09:50:00.344043 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:00.343994    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5d6ddf947f-kmx7m" event={"ID":"0521996d-31be-49c0-84ca-a3ba08d80d71","Type":"ContainerStarted","Data":"e6949d2692fe8c2f2be924d9ae9eacf01ab8eb9fcccaaab52c4b1342a62faa03"}
Sep 14 09:50:02.422934 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:02.422896    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:50:02.537792 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:02.537697    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:50:02.753247 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:02.753206    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:50:02.881622 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:02.881521    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:50:03.077382 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:03.077345    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:50:03.364382 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:03.364342    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5d6ddf947f-kmx7m" event={"ID":"0521996d-31be-49c0-84ca-a3ba08d80d71","Type":"ContainerStarted","Data":"509d204449485ca3912b71ff5f214fe6d52ad6fe395d9de9452ede298da7bfe3"}
Sep 14 09:50:03.364571 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:03.364465    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5d6ddf947f-kmx7m"
Sep 14 09:50:03.387333 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:03.387270    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5d6ddf947f-kmx7m" podStartSLOduration=2.390121242 podStartE2EDuration="4.387244766s" totalImagesPullingTime="1.997123524s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:49:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:50:03.386515987 +0000 UTC m=+392.426190608" watchObservedRunningTime="2026-09-14 09:50:03.387244766 +0000 UTC m=+392.426919402"
Sep 14 09:50:08.337805 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:08.337775    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-rlcpv"
Sep 14 09:50:09.801728 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:09.801688    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:50:14.372072 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:14.372044    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5d6ddf947f-kmx7m"
Sep 14 09:50:20.475550 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:20.475511    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57946b7674-fjvqx"]
Sep 14 09:50:20.600656 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:20.600618    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:50:20.701694 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:20.701661    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:50:20.801780 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:20.801746    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:50:45.495693 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:45.495646    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-57946b7674-fjvqx" podUID="55004416-09dd-42f4-a4e3-520c3e7a0094" containerName="console" containerID="cri-o://17ed7f8133de3ed988bbd11e33f70648c5bdc54f2b716663fa4350ef0ce85c10" gracePeriod=15
Sep 14 09:50:45.744881 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:45.744843    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57946b7674-fjvqx"
Sep 14 09:50:45.826896 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:45.826792    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55004416-09dd-42f4-a4e3-520c3e7a0094-trusted-ca-bundle\") pod \"55004416-09dd-42f4-a4e3-520c3e7a0094\" (UID: \"55004416-09dd-42f4-a4e3-520c3e7a0094\") "
Sep 14 09:50:45.826896 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:45.826857    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/55004416-09dd-42f4-a4e3-520c3e7a0094-console-oauth-config\") pod \"55004416-09dd-42f4-a4e3-520c3e7a0094\" (UID: \"55004416-09dd-42f4-a4e3-520c3e7a0094\") "
Sep 14 09:50:45.826896 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:45.826882    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/55004416-09dd-42f4-a4e3-520c3e7a0094-oauth-serving-cert\") pod \"55004416-09dd-42f4-a4e3-520c3e7a0094\" (UID: \"55004416-09dd-42f4-a4e3-520c3e7a0094\") "
Sep 14 09:50:45.827230 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:45.826927    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4lqgv\" (UniqueName: \"kubernetes.io/projected/55004416-09dd-42f4-a4e3-520c3e7a0094-kube-api-access-4lqgv\") pod \"55004416-09dd-42f4-a4e3-520c3e7a0094\" (UID: \"55004416-09dd-42f4-a4e3-520c3e7a0094\") "
Sep 14 09:50:45.827230 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:45.826995    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/55004416-09dd-42f4-a4e3-520c3e7a0094-service-ca\") pod \"55004416-09dd-42f4-a4e3-520c3e7a0094\" (UID: \"55004416-09dd-42f4-a4e3-520c3e7a0094\") "
Sep 14 09:50:45.827230 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:45.827153    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/55004416-09dd-42f4-a4e3-520c3e7a0094-console-serving-cert\") pod \"55004416-09dd-42f4-a4e3-520c3e7a0094\" (UID: \"55004416-09dd-42f4-a4e3-520c3e7a0094\") "
Sep 14 09:50:45.827230 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:45.827205    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/55004416-09dd-42f4-a4e3-520c3e7a0094-console-config\") pod \"55004416-09dd-42f4-a4e3-520c3e7a0094\" (UID: \"55004416-09dd-42f4-a4e3-520c3e7a0094\") "
Sep 14 09:50:45.827441 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:45.827414    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55004416-09dd-42f4-a4e3-520c3e7a0094-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "55004416-09dd-42f4-a4e3-520c3e7a0094" (UID: "55004416-09dd-42f4-a4e3-520c3e7a0094"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 09:50:45.827441 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:45.827428    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55004416-09dd-42f4-a4e3-520c3e7a0094-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "55004416-09dd-42f4-a4e3-520c3e7a0094" (UID: "55004416-09dd-42f4-a4e3-520c3e7a0094"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 09:50:45.827547 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:45.827441    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55004416-09dd-42f4-a4e3-520c3e7a0094-service-ca" (OuterVolumeSpecName: "service-ca") pod "55004416-09dd-42f4-a4e3-520c3e7a0094" (UID: "55004416-09dd-42f4-a4e3-520c3e7a0094"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 09:50:45.827547 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:45.827509    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/55004416-09dd-42f4-a4e3-520c3e7a0094-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:50:45.827547 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:45.827527    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55004416-09dd-42f4-a4e3-520c3e7a0094-trusted-ca-bundle\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:50:45.827547 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:45.827541    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/55004416-09dd-42f4-a4e3-520c3e7a0094-oauth-serving-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:50:45.827736 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:45.827706    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55004416-09dd-42f4-a4e3-520c3e7a0094-console-config" (OuterVolumeSpecName: "console-config") pod "55004416-09dd-42f4-a4e3-520c3e7a0094" (UID: "55004416-09dd-42f4-a4e3-520c3e7a0094"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 09:50:45.829191 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:45.829161    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55004416-09dd-42f4-a4e3-520c3e7a0094-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "55004416-09dd-42f4-a4e3-520c3e7a0094" (UID: "55004416-09dd-42f4-a4e3-520c3e7a0094"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 09:50:45.829310 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:45.829226    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55004416-09dd-42f4-a4e3-520c3e7a0094-kube-api-access-4lqgv" (OuterVolumeSpecName: "kube-api-access-4lqgv") pod "55004416-09dd-42f4-a4e3-520c3e7a0094" (UID: "55004416-09dd-42f4-a4e3-520c3e7a0094"). InnerVolumeSpecName "kube-api-access-4lqgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 09:50:45.829310 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:45.829247    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55004416-09dd-42f4-a4e3-520c3e7a0094-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "55004416-09dd-42f4-a4e3-520c3e7a0094" (UID: "55004416-09dd-42f4-a4e3-520c3e7a0094"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 09:50:45.928520 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:45.928477    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/55004416-09dd-42f4-a4e3-520c3e7a0094-console-oauth-config\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:50:45.928520 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:45.928509    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lqgv\" (UniqueName: \"kubernetes.io/projected/55004416-09dd-42f4-a4e3-520c3e7a0094-kube-api-access-4lqgv\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:50:45.928520 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:45.928519    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/55004416-09dd-42f4-a4e3-520c3e7a0094-console-serving-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:50:45.928520 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:45.928528    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/55004416-09dd-42f4-a4e3-520c3e7a0094-console-config\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:50:46.572295 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:46.572256    2563 generic.go:356] "Generic (PLEG): container finished" podID="55004416-09dd-42f4-a4e3-520c3e7a0094" containerID="17ed7f8133de3ed988bbd11e33f70648c5bdc54f2b716663fa4350ef0ce85c10" exitCode=0
Sep 14 09:50:46.572750 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:46.572303    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57946b7674-fjvqx"
Sep 14 09:50:46.572750 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:46.572294    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57946b7674-fjvqx" event={"ID":"55004416-09dd-42f4-a4e3-520c3e7a0094","Type":"ContainerDied","Data":"17ed7f8133de3ed988bbd11e33f70648c5bdc54f2b716663fa4350ef0ce85c10"}
Sep 14 09:50:46.572750 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:46.572426    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57946b7674-fjvqx" event={"ID":"55004416-09dd-42f4-a4e3-520c3e7a0094","Type":"ContainerDied","Data":"494e7f0c65a564d5275769b092b50afbab9460c4c09eace4a399ee5ad7ac326c"}
Sep 14 09:50:46.572750 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:46.572447    2563 scope.go:117] "RemoveContainer" containerID="17ed7f8133de3ed988bbd11e33f70648c5bdc54f2b716663fa4350ef0ce85c10"
Sep 14 09:50:46.582374 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:46.582353    2563 scope.go:117] "RemoveContainer" containerID="17ed7f8133de3ed988bbd11e33f70648c5bdc54f2b716663fa4350ef0ce85c10"
Sep 14 09:50:46.582711 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:50:46.582690    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17ed7f8133de3ed988bbd11e33f70648c5bdc54f2b716663fa4350ef0ce85c10\": container with ID starting with 17ed7f8133de3ed988bbd11e33f70648c5bdc54f2b716663fa4350ef0ce85c10 not found: ID does not exist" containerID="17ed7f8133de3ed988bbd11e33f70648c5bdc54f2b716663fa4350ef0ce85c10"
Sep 14 09:50:46.582786 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:46.582718    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17ed7f8133de3ed988bbd11e33f70648c5bdc54f2b716663fa4350ef0ce85c10"} err="failed to get container status \"17ed7f8133de3ed988bbd11e33f70648c5bdc54f2b716663fa4350ef0ce85c10\": rpc error: code = NotFound desc = could not find container \"17ed7f8133de3ed988bbd11e33f70648c5bdc54f2b716663fa4350ef0ce85c10\": container with ID starting with 17ed7f8133de3ed988bbd11e33f70648c5bdc54f2b716663fa4350ef0ce85c10 not found: ID does not exist"
Sep 14 09:50:46.595919 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:46.595883    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57946b7674-fjvqx"]
Sep 14 09:50:46.600714 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:46.600678    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-57946b7674-fjvqx"]
Sep 14 09:50:47.482386 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:47.482349    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55004416-09dd-42f4-a4e3-520c3e7a0094" path="/var/lib/kubelet/pods/55004416-09dd-42f4-a4e3-520c3e7a0094/volumes"
Sep 14 09:50:53.101921 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:53.101880    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:50:53.180454 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:53.180419    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:50:57.312161 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:57.312120    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t5hbp"]
Sep 14 09:50:57.312548 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:57.312468    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55004416-09dd-42f4-a4e3-520c3e7a0094" containerName="console"
Sep 14 09:50:57.312548 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:57.312479    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="55004416-09dd-42f4-a4e3-520c3e7a0094" containerName="console"
Sep 14 09:50:57.312548 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:57.312530    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="55004416-09dd-42f4-a4e3-520c3e7a0094" containerName="console"
Sep 14 09:50:57.314805 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:57.314783    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t5hbp"
Sep 14 09:50:57.317124 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:57.317094    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 14 09:50:57.317307 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:57.317188    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 14 09:50:57.317863 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:57.317838    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-4r7sj\""
Sep 14 09:50:57.323470 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:57.323441    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t5hbp"]
Sep 14 09:50:57.411721 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:57.411680    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmwfs6"]
Sep 14 09:50:57.414549 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:57.414522    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmwfs6"
Sep 14 09:50:57.424170 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:57.424129    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmwfs6"]
Sep 14 09:50:57.432229 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:57.432190    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4f1341e9-56a0-4e70-b02d-3c0fbac5f23a-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t5hbp\" (UID: \"4f1341e9-56a0-4e70-b02d-3c0fbac5f23a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t5hbp"
Sep 14 09:50:57.432401 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:57.432280    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhx2z\" (UniqueName: \"kubernetes.io/projected/4f1341e9-56a0-4e70-b02d-3c0fbac5f23a-kube-api-access-vhx2z\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t5hbp\" (UID: \"4f1341e9-56a0-4e70-b02d-3c0fbac5f23a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t5hbp"
Sep 14 09:50:57.432401 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:57.432305    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4f1341e9-56a0-4e70-b02d-3c0fbac5f23a-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t5hbp\" (UID: \"4f1341e9-56a0-4e70-b02d-3c0fbac5f23a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t5hbp"
Sep 14 09:50:57.511194 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:57.511147    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7srzfv"]
Sep 14 09:50:57.513774 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:57.513745    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7srzfv"
Sep 14 09:50:57.523861 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:57.523826    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7srzfv"]
Sep 14 09:50:57.533410 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:57.533367    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8g68\" (UniqueName: \"kubernetes.io/projected/fee8294a-efe0-43c7-92f3-48f39e86306e-kube-api-access-h8g68\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmwfs6\" (UID: \"fee8294a-efe0-43c7-92f3-48f39e86306e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmwfs6"
Sep 14 09:50:57.533617 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:57.533439    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhx2z\" (UniqueName: \"kubernetes.io/projected/4f1341e9-56a0-4e70-b02d-3c0fbac5f23a-kube-api-access-vhx2z\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t5hbp\" (UID: \"4f1341e9-56a0-4e70-b02d-3c0fbac5f23a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t5hbp"
Sep 14 09:50:57.533617 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:57.533512    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4f1341e9-56a0-4e70-b02d-3c0fbac5f23a-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t5hbp\" (UID: \"4f1341e9-56a0-4e70-b02d-3c0fbac5f23a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t5hbp"
Sep 14 09:50:57.533617 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:57.533560    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fee8294a-efe0-43c7-92f3-48f39e86306e-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmwfs6\" (UID: \"fee8294a-efe0-43c7-92f3-48f39e86306e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmwfs6"
Sep 14 09:50:57.533617 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:57.533608    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fee8294a-efe0-43c7-92f3-48f39e86306e-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmwfs6\" (UID: \"fee8294a-efe0-43c7-92f3-48f39e86306e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmwfs6"
Sep 14 09:50:57.533803 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:57.533642    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4f1341e9-56a0-4e70-b02d-3c0fbac5f23a-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t5hbp\" (UID: \"4f1341e9-56a0-4e70-b02d-3c0fbac5f23a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t5hbp"
Sep 14 09:50:57.533907 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:57.533888    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4f1341e9-56a0-4e70-b02d-3c0fbac5f23a-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t5hbp\" (UID: \"4f1341e9-56a0-4e70-b02d-3c0fbac5f23a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t5hbp"
Sep 14 09:50:57.533995 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:57.533975    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4f1341e9-56a0-4e70-b02d-3c0fbac5f23a-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t5hbp\" (UID: \"4f1341e9-56a0-4e70-b02d-3c0fbac5f23a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t5hbp"
Sep 14 09:50:57.544653 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:57.544616    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhx2z\" (UniqueName: \"kubernetes.io/projected/4f1341e9-56a0-4e70-b02d-3c0fbac5f23a-kube-api-access-vhx2z\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t5hbp\" (UID: \"4f1341e9-56a0-4e70-b02d-3c0fbac5f23a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t5hbp"
Sep 14 09:50:57.616492 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:57.616400    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847artxm6"]
Sep 14 09:50:57.619160 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:57.619132    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847artxm6"
Sep 14 09:50:57.625332 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:57.625301    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t5hbp"
Sep 14 09:50:57.630965 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:57.630931    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847artxm6"]
Sep 14 09:50:57.636918 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:57.636881    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5bc0e2ad-2b0e-4f8e-bae8-0cfc03237963-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7srzfv\" (UID: \"5bc0e2ad-2b0e-4f8e-bae8-0cfc03237963\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7srzfv"
Sep 14 09:50:57.636918 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:57.636920    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5bc0e2ad-2b0e-4f8e-bae8-0cfc03237963-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7srzfv\" (UID: \"5bc0e2ad-2b0e-4f8e-bae8-0cfc03237963\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7srzfv"
Sep 14 09:50:57.637185 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:57.636955    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8g68\" (UniqueName: \"kubernetes.io/projected/fee8294a-efe0-43c7-92f3-48f39e86306e-kube-api-access-h8g68\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmwfs6\" (UID: \"fee8294a-efe0-43c7-92f3-48f39e86306e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmwfs6"
Sep 14 09:50:57.637185 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:57.636988    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtrpx\" (UniqueName: \"kubernetes.io/projected/5bc0e2ad-2b0e-4f8e-bae8-0cfc03237963-kube-api-access-rtrpx\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7srzfv\" (UID: \"5bc0e2ad-2b0e-4f8e-bae8-0cfc03237963\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7srzfv"
Sep 14 09:50:57.637185 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:57.637106    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fee8294a-efe0-43c7-92f3-48f39e86306e-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmwfs6\" (UID: \"fee8294a-efe0-43c7-92f3-48f39e86306e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmwfs6"
Sep 14 09:50:57.637185 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:57.637155    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fee8294a-efe0-43c7-92f3-48f39e86306e-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmwfs6\" (UID: \"fee8294a-efe0-43c7-92f3-48f39e86306e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmwfs6"
Sep 14 09:50:57.637620 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:57.637592    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fee8294a-efe0-43c7-92f3-48f39e86306e-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmwfs6\" (UID: \"fee8294a-efe0-43c7-92f3-48f39e86306e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmwfs6"
Sep 14 09:50:57.637770 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:57.637694    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fee8294a-efe0-43c7-92f3-48f39e86306e-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmwfs6\" (UID: \"fee8294a-efe0-43c7-92f3-48f39e86306e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmwfs6"
Sep 14 09:50:57.646236 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:57.646205    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8g68\" (UniqueName: \"kubernetes.io/projected/fee8294a-efe0-43c7-92f3-48f39e86306e-kube-api-access-h8g68\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmwfs6\" (UID: \"fee8294a-efe0-43c7-92f3-48f39e86306e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmwfs6"
Sep 14 09:50:57.725976 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:57.725937    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmwfs6"
Sep 14 09:50:57.738469 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:57.738431    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgwxp\" (UniqueName: \"kubernetes.io/projected/4e848ae4-643a-4874-9e63-1a41fdf5288d-kube-api-access-tgwxp\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847artxm6\" (UID: \"4e848ae4-643a-4874-9e63-1a41fdf5288d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847artxm6"
Sep 14 09:50:57.738616 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:57.738479    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5bc0e2ad-2b0e-4f8e-bae8-0cfc03237963-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7srzfv\" (UID: \"5bc0e2ad-2b0e-4f8e-bae8-0cfc03237963\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7srzfv"
Sep 14 09:50:57.738616 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:57.738496    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5bc0e2ad-2b0e-4f8e-bae8-0cfc03237963-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7srzfv\" (UID: \"5bc0e2ad-2b0e-4f8e-bae8-0cfc03237963\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7srzfv"
Sep 14 09:50:57.738616 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:57.738562    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4e848ae4-643a-4874-9e63-1a41fdf5288d-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847artxm6\" (UID: \"4e848ae4-643a-4874-9e63-1a41fdf5288d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847artxm6"
Sep 14 09:50:57.738797 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:57.738607    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtrpx\" (UniqueName: \"kubernetes.io/projected/5bc0e2ad-2b0e-4f8e-bae8-0cfc03237963-kube-api-access-rtrpx\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7srzfv\" (UID: \"5bc0e2ad-2b0e-4f8e-bae8-0cfc03237963\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7srzfv"
Sep 14 09:50:57.738797 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:57.738703    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4e848ae4-643a-4874-9e63-1a41fdf5288d-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847artxm6\" (UID: \"4e848ae4-643a-4874-9e63-1a41fdf5288d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847artxm6"
Sep 14 09:50:57.738961 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:57.738817    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5bc0e2ad-2b0e-4f8e-bae8-0cfc03237963-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7srzfv\" (UID: \"5bc0e2ad-2b0e-4f8e-bae8-0cfc03237963\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7srzfv"
Sep 14 09:50:57.738961 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:57.738879    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5bc0e2ad-2b0e-4f8e-bae8-0cfc03237963-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7srzfv\" (UID: \"5bc0e2ad-2b0e-4f8e-bae8-0cfc03237963\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7srzfv"
Sep 14 09:50:57.748421 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:57.748376    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtrpx\" (UniqueName: \"kubernetes.io/projected/5bc0e2ad-2b0e-4f8e-bae8-0cfc03237963-kube-api-access-rtrpx\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7srzfv\" (UID: \"5bc0e2ad-2b0e-4f8e-bae8-0cfc03237963\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7srzfv"
Sep 14 09:50:57.824701 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:57.824659    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7srzfv"
Sep 14 09:50:57.839481 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:57.839442    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4e848ae4-643a-4874-9e63-1a41fdf5288d-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847artxm6\" (UID: \"4e848ae4-643a-4874-9e63-1a41fdf5288d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847artxm6"
Sep 14 09:50:57.839683 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:57.839526    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgwxp\" (UniqueName: \"kubernetes.io/projected/4e848ae4-643a-4874-9e63-1a41fdf5288d-kube-api-access-tgwxp\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847artxm6\" (UID: \"4e848ae4-643a-4874-9e63-1a41fdf5288d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847artxm6"
Sep 14 09:50:57.839683 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:57.839564    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4e848ae4-643a-4874-9e63-1a41fdf5288d-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847artxm6\" (UID: \"4e848ae4-643a-4874-9e63-1a41fdf5288d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847artxm6"
Sep 14 09:50:57.839963 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:57.839937    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4e848ae4-643a-4874-9e63-1a41fdf5288d-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847artxm6\" (UID: \"4e848ae4-643a-4874-9e63-1a41fdf5288d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847artxm6"
Sep 14 09:50:57.840123 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:57.839949    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4e848ae4-643a-4874-9e63-1a41fdf5288d-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847artxm6\" (UID: \"4e848ae4-643a-4874-9e63-1a41fdf5288d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847artxm6"
Sep 14 09:50:57.848606 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:57.848568    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgwxp\" (UniqueName: \"kubernetes.io/projected/4e848ae4-643a-4874-9e63-1a41fdf5288d-kube-api-access-tgwxp\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847artxm6\" (UID: \"4e848ae4-643a-4874-9e63-1a41fdf5288d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847artxm6"
Sep 14 09:50:57.913166 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:57.913130    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t5hbp"]
Sep 14 09:50:57.914773 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:50:57.914745    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f1341e9_56a0_4e70_b02d_3c0fbac5f23a.slice/crio-44971f6218f139f1e3fc47a0d703eb7e681cd71ae0fce287d1eb007ccdcf1e5c WatchSource:0}: Error finding container 44971f6218f139f1e3fc47a0d703eb7e681cd71ae0fce287d1eb007ccdcf1e5c: Status 404 returned error can't find the container with id 44971f6218f139f1e3fc47a0d703eb7e681cd71ae0fce287d1eb007ccdcf1e5c
Sep 14 09:50:57.932500 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:57.932471    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847artxm6"
Sep 14 09:50:58.011175 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:58.011123    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmwfs6"]
Sep 14 09:50:58.014907 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:50:58.014865    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfee8294a_efe0_43c7_92f3_48f39e86306e.slice/crio-1fca11d6d656899e4da938cc3d9f649cf06b6a04bd6b7063295ac9c110cae53b WatchSource:0}: Error finding container 1fca11d6d656899e4da938cc3d9f649cf06b6a04bd6b7063295ac9c110cae53b: Status 404 returned error can't find the container with id 1fca11d6d656899e4da938cc3d9f649cf06b6a04bd6b7063295ac9c110cae53b
Sep 14 09:50:58.102924 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:58.102888    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7srzfv"]
Sep 14 09:50:58.104817 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:50:58.104774    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bc0e2ad_2b0e_4f8e_bae8_0cfc03237963.slice/crio-0ec7ac10ede021091f33b17a50a09f6c5a44b72e4d79c56d3d68a19584b7bf62 WatchSource:0}: Error finding container 0ec7ac10ede021091f33b17a50a09f6c5a44b72e4d79c56d3d68a19584b7bf62: Status 404 returned error can't find the container with id 0ec7ac10ede021091f33b17a50a09f6c5a44b72e4d79c56d3d68a19584b7bf62
Sep 14 09:50:58.213209 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:58.213174    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847artxm6"]
Sep 14 09:50:58.260131 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:50:58.260093    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e848ae4_643a_4874_9e63_1a41fdf5288d.slice/crio-e5f704125b942cdb73c0f6bfa629d511187334fada0a8ed39d9922c05b90de73 WatchSource:0}: Error finding container e5f704125b942cdb73c0f6bfa629d511187334fada0a8ed39d9922c05b90de73: Status 404 returned error can't find the container with id e5f704125b942cdb73c0f6bfa629d511187334fada0a8ed39d9922c05b90de73
Sep 14 09:50:58.637204 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:58.637158    2563 generic.go:356] "Generic (PLEG): container finished" podID="5bc0e2ad-2b0e-4f8e-bae8-0cfc03237963" containerID="9b76ffc558cd3bc877f67b9eea8dd7f02b44d6f5a01eafebb6d415aa1d32aeb7" exitCode=0
Sep 14 09:50:58.637672 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:58.637239    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7srzfv" event={"ID":"5bc0e2ad-2b0e-4f8e-bae8-0cfc03237963","Type":"ContainerDied","Data":"9b76ffc558cd3bc877f67b9eea8dd7f02b44d6f5a01eafebb6d415aa1d32aeb7"}
Sep 14 09:50:58.637672 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:58.637289    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7srzfv" event={"ID":"5bc0e2ad-2b0e-4f8e-bae8-0cfc03237963","Type":"ContainerStarted","Data":"0ec7ac10ede021091f33b17a50a09f6c5a44b72e4d79c56d3d68a19584b7bf62"}
Sep 14 09:50:58.639083 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:58.639055    2563 generic.go:356] "Generic (PLEG): container finished" podID="4e848ae4-643a-4874-9e63-1a41fdf5288d" containerID="749281fadc6c272e5df224001fd2e3efe0b3f0d9e5df0a83c062b77637cef822" exitCode=0
Sep 14 09:50:58.639194 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:58.639142    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847artxm6" event={"ID":"4e848ae4-643a-4874-9e63-1a41fdf5288d","Type":"ContainerDied","Data":"749281fadc6c272e5df224001fd2e3efe0b3f0d9e5df0a83c062b77637cef822"}
Sep 14 09:50:58.639259 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:58.639199    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847artxm6" event={"ID":"4e848ae4-643a-4874-9e63-1a41fdf5288d","Type":"ContainerStarted","Data":"e5f704125b942cdb73c0f6bfa629d511187334fada0a8ed39d9922c05b90de73"}
Sep 14 09:50:58.640818 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:58.640793    2563 generic.go:356] "Generic (PLEG): container finished" podID="fee8294a-efe0-43c7-92f3-48f39e86306e" containerID="cbfe6288575de5bdee89707a4deae80fde0d4fd34c064f4057645d4553318132" exitCode=0
Sep 14 09:50:58.640932 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:58.640860    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmwfs6" event={"ID":"fee8294a-efe0-43c7-92f3-48f39e86306e","Type":"ContainerDied","Data":"cbfe6288575de5bdee89707a4deae80fde0d4fd34c064f4057645d4553318132"}
Sep 14 09:50:58.640932 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:58.640925    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmwfs6" event={"ID":"fee8294a-efe0-43c7-92f3-48f39e86306e","Type":"ContainerStarted","Data":"1fca11d6d656899e4da938cc3d9f649cf06b6a04bd6b7063295ac9c110cae53b"}
Sep 14 09:50:58.642635 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:58.642608    2563 generic.go:356] "Generic (PLEG): container finished" podID="4f1341e9-56a0-4e70-b02d-3c0fbac5f23a" containerID="4db429995ee08a4051c5a82a4732cd2f94db85f25c00195cc57eb07cc0fa9f23" exitCode=0
Sep 14 09:50:58.642781 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:58.642660    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t5hbp" event={"ID":"4f1341e9-56a0-4e70-b02d-3c0fbac5f23a","Type":"ContainerDied","Data":"4db429995ee08a4051c5a82a4732cd2f94db85f25c00195cc57eb07cc0fa9f23"}
Sep 14 09:50:58.642781 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:58.642691    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t5hbp" event={"ID":"4f1341e9-56a0-4e70-b02d-3c0fbac5f23a","Type":"ContainerStarted","Data":"44971f6218f139f1e3fc47a0d703eb7e681cd71ae0fce287d1eb007ccdcf1e5c"}
Sep 14 09:50:59.650139 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:59.650099    2563 generic.go:356] "Generic (PLEG): container finished" podID="fee8294a-efe0-43c7-92f3-48f39e86306e" containerID="e4863ff69e7fc5cb06f9953a7fb17431c6d395f939f2f8af89ee738b51d30cfe" exitCode=0
Sep 14 09:50:59.650623 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:59.650185    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmwfs6" event={"ID":"fee8294a-efe0-43c7-92f3-48f39e86306e","Type":"ContainerDied","Data":"e4863ff69e7fc5cb06f9953a7fb17431c6d395f939f2f8af89ee738b51d30cfe"}
Sep 14 09:50:59.652290 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:59.652253    2563 generic.go:356] "Generic (PLEG): container finished" podID="4f1341e9-56a0-4e70-b02d-3c0fbac5f23a" containerID="6a9f66e186eedd5ac3856fdc5c4a6be26c8774ef5775941fca4e7e2f31a4b2df" exitCode=0
Sep 14 09:50:59.652399 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:50:59.652308    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t5hbp" event={"ID":"4f1341e9-56a0-4e70-b02d-3c0fbac5f23a","Type":"ContainerDied","Data":"6a9f66e186eedd5ac3856fdc5c4a6be26c8774ef5775941fca4e7e2f31a4b2df"}
Sep 14 09:51:00.660087 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:00.660041    2563 generic.go:356] "Generic (PLEG): container finished" podID="fee8294a-efe0-43c7-92f3-48f39e86306e" containerID="1ef1e366f3dfa575fd96b6f8e4e73be0b3bc823f8adb41fbc0198579602a4532" exitCode=0
Sep 14 09:51:00.660501 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:00.660129    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmwfs6" event={"ID":"fee8294a-efe0-43c7-92f3-48f39e86306e","Type":"ContainerDied","Data":"1ef1e366f3dfa575fd96b6f8e4e73be0b3bc823f8adb41fbc0198579602a4532"}
Sep 14 09:51:00.662250 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:00.662221    2563 generic.go:356] "Generic (PLEG): container finished" podID="4f1341e9-56a0-4e70-b02d-3c0fbac5f23a" containerID="f0f4fd1ac4edff1b7f0c03bcaad91722c4ae479c33cb99c8e65a1658142a992f" exitCode=0
Sep 14 09:51:00.662401 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:00.662281    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t5hbp" event={"ID":"4f1341e9-56a0-4e70-b02d-3c0fbac5f23a","Type":"ContainerDied","Data":"f0f4fd1ac4edff1b7f0c03bcaad91722c4ae479c33cb99c8e65a1658142a992f"}
Sep 14 09:51:01.670683 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:01.670587    2563 generic.go:356] "Generic (PLEG): container finished" podID="5bc0e2ad-2b0e-4f8e-bae8-0cfc03237963" containerID="b7d2703f5ec5392d880837b4a570cbd0aed8292aef0f533b49bec3a00da9039e" exitCode=0
Sep 14 09:51:01.670683 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:01.670669    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7srzfv" event={"ID":"5bc0e2ad-2b0e-4f8e-bae8-0cfc03237963","Type":"ContainerDied","Data":"b7d2703f5ec5392d880837b4a570cbd0aed8292aef0f533b49bec3a00da9039e"}
Sep 14 09:51:01.894830 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:01.894797    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t5hbp"
Sep 14 09:51:01.898400 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:01.898368    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmwfs6"
Sep 14 09:51:01.976605 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:01.976508    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vhx2z\" (UniqueName: \"kubernetes.io/projected/4f1341e9-56a0-4e70-b02d-3c0fbac5f23a-kube-api-access-vhx2z\") pod \"4f1341e9-56a0-4e70-b02d-3c0fbac5f23a\" (UID: \"4f1341e9-56a0-4e70-b02d-3c0fbac5f23a\") "
Sep 14 09:51:01.976605 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:01.976550    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fee8294a-efe0-43c7-92f3-48f39e86306e-util\") pod \"fee8294a-efe0-43c7-92f3-48f39e86306e\" (UID: \"fee8294a-efe0-43c7-92f3-48f39e86306e\") "
Sep 14 09:51:01.976846 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:01.976614    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4f1341e9-56a0-4e70-b02d-3c0fbac5f23a-bundle\") pod \"4f1341e9-56a0-4e70-b02d-3c0fbac5f23a\" (UID: \"4f1341e9-56a0-4e70-b02d-3c0fbac5f23a\") "
Sep 14 09:51:01.976846 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:01.976644    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h8g68\" (UniqueName: \"kubernetes.io/projected/fee8294a-efe0-43c7-92f3-48f39e86306e-kube-api-access-h8g68\") pod \"fee8294a-efe0-43c7-92f3-48f39e86306e\" (UID: \"fee8294a-efe0-43c7-92f3-48f39e86306e\") "
Sep 14 09:51:01.976846 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:01.976668    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4f1341e9-56a0-4e70-b02d-3c0fbac5f23a-util\") pod \"4f1341e9-56a0-4e70-b02d-3c0fbac5f23a\" (UID: \"4f1341e9-56a0-4e70-b02d-3c0fbac5f23a\") "
Sep 14 09:51:01.976846 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:01.976686    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fee8294a-efe0-43c7-92f3-48f39e86306e-bundle\") pod \"fee8294a-efe0-43c7-92f3-48f39e86306e\" (UID: \"fee8294a-efe0-43c7-92f3-48f39e86306e\") "
Sep 14 09:51:01.977133 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:01.977105    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f1341e9-56a0-4e70-b02d-3c0fbac5f23a-bundle" (OuterVolumeSpecName: "bundle") pod "4f1341e9-56a0-4e70-b02d-3c0fbac5f23a" (UID: "4f1341e9-56a0-4e70-b02d-3c0fbac5f23a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 09:51:01.977384 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:01.977353    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fee8294a-efe0-43c7-92f3-48f39e86306e-bundle" (OuterVolumeSpecName: "bundle") pod "fee8294a-efe0-43c7-92f3-48f39e86306e" (UID: "fee8294a-efe0-43c7-92f3-48f39e86306e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 09:51:01.978987 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:01.978960    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fee8294a-efe0-43c7-92f3-48f39e86306e-kube-api-access-h8g68" (OuterVolumeSpecName: "kube-api-access-h8g68") pod "fee8294a-efe0-43c7-92f3-48f39e86306e" (UID: "fee8294a-efe0-43c7-92f3-48f39e86306e"). InnerVolumeSpecName "kube-api-access-h8g68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 09:51:01.979111 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:01.978996    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f1341e9-56a0-4e70-b02d-3c0fbac5f23a-kube-api-access-vhx2z" (OuterVolumeSpecName: "kube-api-access-vhx2z") pod "4f1341e9-56a0-4e70-b02d-3c0fbac5f23a" (UID: "4f1341e9-56a0-4e70-b02d-3c0fbac5f23a"). InnerVolumeSpecName "kube-api-access-vhx2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 09:51:01.982368 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:01.982316    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f1341e9-56a0-4e70-b02d-3c0fbac5f23a-util" (OuterVolumeSpecName: "util") pod "4f1341e9-56a0-4e70-b02d-3c0fbac5f23a" (UID: "4f1341e9-56a0-4e70-b02d-3c0fbac5f23a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 09:51:01.982871 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:01.982848    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fee8294a-efe0-43c7-92f3-48f39e86306e-util" (OuterVolumeSpecName: "util") pod "fee8294a-efe0-43c7-92f3-48f39e86306e" (UID: "fee8294a-efe0-43c7-92f3-48f39e86306e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 09:51:02.077758 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:02.077719    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhx2z\" (UniqueName: \"kubernetes.io/projected/4f1341e9-56a0-4e70-b02d-3c0fbac5f23a-kube-api-access-vhx2z\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:51:02.077758 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:02.077751    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fee8294a-efe0-43c7-92f3-48f39e86306e-util\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:51:02.077758 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:02.077764    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4f1341e9-56a0-4e70-b02d-3c0fbac5f23a-bundle\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:51:02.078040 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:02.077776    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8g68\" (UniqueName: \"kubernetes.io/projected/fee8294a-efe0-43c7-92f3-48f39e86306e-kube-api-access-h8g68\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:51:02.078040 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:02.077789    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4f1341e9-56a0-4e70-b02d-3c0fbac5f23a-util\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:51:02.078040 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:02.077802    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fee8294a-efe0-43c7-92f3-48f39e86306e-bundle\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:51:02.679671 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:02.679582    2563 generic.go:356] "Generic (PLEG): container finished" podID="5bc0e2ad-2b0e-4f8e-bae8-0cfc03237963" containerID="23854ead263b0c7a4712f2bc9438be5e9e682241e6733f93480d7e8bcecc6f2a" exitCode=0
Sep 14 09:51:02.680143 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:02.679665    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7srzfv" event={"ID":"5bc0e2ad-2b0e-4f8e-bae8-0cfc03237963","Type":"ContainerDied","Data":"23854ead263b0c7a4712f2bc9438be5e9e682241e6733f93480d7e8bcecc6f2a"}
Sep 14 09:51:02.681839 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:02.681818    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmwfs6"
Sep 14 09:51:02.681974 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:02.681818    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmwfs6" event={"ID":"fee8294a-efe0-43c7-92f3-48f39e86306e","Type":"ContainerDied","Data":"1fca11d6d656899e4da938cc3d9f649cf06b6a04bd6b7063295ac9c110cae53b"}
Sep 14 09:51:02.681974 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:02.681937    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1fca11d6d656899e4da938cc3d9f649cf06b6a04bd6b7063295ac9c110cae53b"
Sep 14 09:51:02.684098 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:02.684055    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t5hbp" event={"ID":"4f1341e9-56a0-4e70-b02d-3c0fbac5f23a","Type":"ContainerDied","Data":"44971f6218f139f1e3fc47a0d703eb7e681cd71ae0fce287d1eb007ccdcf1e5c"}
Sep 14 09:51:02.684221 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:02.684110    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44971f6218f139f1e3fc47a0d703eb7e681cd71ae0fce287d1eb007ccdcf1e5c"
Sep 14 09:51:02.684221 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:02.684075    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t5hbp"
Sep 14 09:51:03.824110 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:03.824073    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7srzfv"
Sep 14 09:51:03.892480 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:03.892439    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5bc0e2ad-2b0e-4f8e-bae8-0cfc03237963-bundle\") pod \"5bc0e2ad-2b0e-4f8e-bae8-0cfc03237963\" (UID: \"5bc0e2ad-2b0e-4f8e-bae8-0cfc03237963\") "
Sep 14 09:51:03.892480 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:03.892489    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5bc0e2ad-2b0e-4f8e-bae8-0cfc03237963-util\") pod \"5bc0e2ad-2b0e-4f8e-bae8-0cfc03237963\" (UID: \"5bc0e2ad-2b0e-4f8e-bae8-0cfc03237963\") "
Sep 14 09:51:03.892734 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:03.892612    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rtrpx\" (UniqueName: \"kubernetes.io/projected/5bc0e2ad-2b0e-4f8e-bae8-0cfc03237963-kube-api-access-rtrpx\") pod \"5bc0e2ad-2b0e-4f8e-bae8-0cfc03237963\" (UID: \"5bc0e2ad-2b0e-4f8e-bae8-0cfc03237963\") "
Sep 14 09:51:03.892962 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:03.892936    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bc0e2ad-2b0e-4f8e-bae8-0cfc03237963-bundle" (OuterVolumeSpecName: "bundle") pod "5bc0e2ad-2b0e-4f8e-bae8-0cfc03237963" (UID: "5bc0e2ad-2b0e-4f8e-bae8-0cfc03237963"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 09:51:03.894742 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:03.894712    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bc0e2ad-2b0e-4f8e-bae8-0cfc03237963-kube-api-access-rtrpx" (OuterVolumeSpecName: "kube-api-access-rtrpx") pod "5bc0e2ad-2b0e-4f8e-bae8-0cfc03237963" (UID: "5bc0e2ad-2b0e-4f8e-bae8-0cfc03237963"). InnerVolumeSpecName "kube-api-access-rtrpx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 09:51:03.897215 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:03.897182    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bc0e2ad-2b0e-4f8e-bae8-0cfc03237963-util" (OuterVolumeSpecName: "util") pod "5bc0e2ad-2b0e-4f8e-bae8-0cfc03237963" (UID: "5bc0e2ad-2b0e-4f8e-bae8-0cfc03237963"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 09:51:03.993921 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:03.993884    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtrpx\" (UniqueName: \"kubernetes.io/projected/5bc0e2ad-2b0e-4f8e-bae8-0cfc03237963-kube-api-access-rtrpx\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:51:03.993921 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:03.993918    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5bc0e2ad-2b0e-4f8e-bae8-0cfc03237963-bundle\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:51:03.993921 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:03.993927    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5bc0e2ad-2b0e-4f8e-bae8-0cfc03237963-util\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:51:04.696825 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:04.696789    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7srzfv"
Sep 14 09:51:04.696825 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:04.696807    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7srzfv" event={"ID":"5bc0e2ad-2b0e-4f8e-bae8-0cfc03237963","Type":"ContainerDied","Data":"0ec7ac10ede021091f33b17a50a09f6c5a44b72e4d79c56d3d68a19584b7bf62"}
Sep 14 09:51:04.697059 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:04.696860    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ec7ac10ede021091f33b17a50a09f6c5a44b72e4d79c56d3d68a19584b7bf62"
Sep 14 09:51:08.719747 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:08.719704    2563 generic.go:356] "Generic (PLEG): container finished" podID="4e848ae4-643a-4874-9e63-1a41fdf5288d" containerID="a04a4e4fd7a0b9653afce6766963a697d850ce8095cda00ce754a7def288d8bd" exitCode=0
Sep 14 09:51:08.720295 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:08.719789    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847artxm6" event={"ID":"4e848ae4-643a-4874-9e63-1a41fdf5288d","Type":"ContainerDied","Data":"a04a4e4fd7a0b9653afce6766963a697d850ce8095cda00ce754a7def288d8bd"}
Sep 14 09:51:09.733257 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:09.733217    2563 generic.go:356] "Generic (PLEG): container finished" podID="4e848ae4-643a-4874-9e63-1a41fdf5288d" containerID="d131c66c0baf79a3c5535bfaa4efcec8c527fcd928a1e4594341a5a2117082e4" exitCode=0
Sep 14 09:51:09.733682 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:09.733301    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847artxm6" event={"ID":"4e848ae4-643a-4874-9e63-1a41fdf5288d","Type":"ContainerDied","Data":"d131c66c0baf79a3c5535bfaa4efcec8c527fcd928a1e4594341a5a2117082e4"}
Sep 14 09:51:10.872641 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:10.872607    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847artxm6"
Sep 14 09:51:10.956766 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:10.956725    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4e848ae4-643a-4874-9e63-1a41fdf5288d-bundle\") pod \"4e848ae4-643a-4874-9e63-1a41fdf5288d\" (UID: \"4e848ae4-643a-4874-9e63-1a41fdf5288d\") "
Sep 14 09:51:10.956974 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:10.956783    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tgwxp\" (UniqueName: \"kubernetes.io/projected/4e848ae4-643a-4874-9e63-1a41fdf5288d-kube-api-access-tgwxp\") pod \"4e848ae4-643a-4874-9e63-1a41fdf5288d\" (UID: \"4e848ae4-643a-4874-9e63-1a41fdf5288d\") "
Sep 14 09:51:10.956974 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:10.956812    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4e848ae4-643a-4874-9e63-1a41fdf5288d-util\") pod \"4e848ae4-643a-4874-9e63-1a41fdf5288d\" (UID: \"4e848ae4-643a-4874-9e63-1a41fdf5288d\") "
Sep 14 09:51:10.957531 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:10.957497    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e848ae4-643a-4874-9e63-1a41fdf5288d-bundle" (OuterVolumeSpecName: "bundle") pod "4e848ae4-643a-4874-9e63-1a41fdf5288d" (UID: "4e848ae4-643a-4874-9e63-1a41fdf5288d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 09:51:10.958952 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:10.958919    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e848ae4-643a-4874-9e63-1a41fdf5288d-kube-api-access-tgwxp" (OuterVolumeSpecName: "kube-api-access-tgwxp") pod "4e848ae4-643a-4874-9e63-1a41fdf5288d" (UID: "4e848ae4-643a-4874-9e63-1a41fdf5288d"). InnerVolumeSpecName "kube-api-access-tgwxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 09:51:10.961328 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:10.961299    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e848ae4-643a-4874-9e63-1a41fdf5288d-util" (OuterVolumeSpecName: "util") pod "4e848ae4-643a-4874-9e63-1a41fdf5288d" (UID: "4e848ae4-643a-4874-9e63-1a41fdf5288d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 09:51:11.058448 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:11.058354    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4e848ae4-643a-4874-9e63-1a41fdf5288d-bundle\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:51:11.058448 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:11.058385    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgwxp\" (UniqueName: \"kubernetes.io/projected/4e848ae4-643a-4874-9e63-1a41fdf5288d-kube-api-access-tgwxp\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:51:11.058448 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:11.058396    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4e848ae4-643a-4874-9e63-1a41fdf5288d-util\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:51:11.745479 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:11.745387    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847artxm6" event={"ID":"4e848ae4-643a-4874-9e63-1a41fdf5288d","Type":"ContainerDied","Data":"e5f704125b942cdb73c0f6bfa629d511187334fada0a8ed39d9922c05b90de73"}
Sep 14 09:51:11.745479 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:11.745432    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847artxm6"
Sep 14 09:51:11.745479 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:11.745445    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5f704125b942cdb73c0f6bfa629d511187334fada0a8ed39d9922c05b90de73"
Sep 14 09:51:15.968566 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:15.968510    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:51:18.576114 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:18.576073    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:51:18.970854 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:18.970762    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:51:19.417879 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:19.417841    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:51:22.341810 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:22.341771    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:51:22.969727 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:22.969694    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:51:23.356170 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:23.356137    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:51:23.457446 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:23.457411    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:51:24.587783 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:24.587744    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:51:24.826561 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:24.826518    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:51:24.976580 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:24.976500    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:51:25.093135 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:25.093102    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:51:26.216167 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:26.216130    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:51:28.577487 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:28.577451    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:51:28.696312 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:28.696277    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:51:29.672042 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:29.671993    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:51:30.021545 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:30.021508    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:51:32.607676 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:32.607641    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:51:32.699922 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:32.699886    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:51:32.972636 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:32.972545    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:51:33.078613 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:33.078561    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:51:33.182271 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:33.182239    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:51:33.282106 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:33.282073    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:51:35.082816 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:35.082778    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:51:38.856970 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:38.856934    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:51:38.972472 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:38.972435    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:51:42.072346 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:42.072308    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:51:42.173690 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:42.173657    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:51:45.267408 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:45.267370    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:51:45.366981 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:45.366945    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:51:47.888424 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:47.888383    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:51:47.993183 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:47.993147    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:51:51.682617 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:51.682583    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:51:51.784653 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:51.784621    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:51:51.886222 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:51.886186    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:51:51.990919 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:51.990887    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:51:53.011927 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:53.011891    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:51:53.105834 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:53.105802    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:51:56.651882 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:56.651851    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:51:56.754408 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:56.754372    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:51:59.744263 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:59.744220    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:51:59.843352 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:51:59.843312    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:52:06.300508 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:06.300470    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:52:06.399391 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:06.399351    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:52:11.049145 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:11.049105    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:52:11.150751 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:11.150715    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:52:12.607751 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:12.607709    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:52:12.710610 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:12.710577    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:52:18.017571 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:18.017533    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:52:18.102895 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:18.102861    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:52:18.208308 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:18.208271    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:52:18.806321 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:18.806275    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-zqkdt"]
Sep 14 09:52:18.806707 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:18.806688    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fee8294a-efe0-43c7-92f3-48f39e86306e" containerName="util"
Sep 14 09:52:18.806806 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:18.806710    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fee8294a-efe0-43c7-92f3-48f39e86306e" containerName="util"
Sep 14 09:52:18.806806 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:18.806722    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fee8294a-efe0-43c7-92f3-48f39e86306e" containerName="extract"
Sep 14 09:52:18.806806 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:18.806732    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fee8294a-efe0-43c7-92f3-48f39e86306e" containerName="extract"
Sep 14 09:52:18.806806 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:18.806741    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f1341e9-56a0-4e70-b02d-3c0fbac5f23a" containerName="extract"
Sep 14 09:52:18.806806 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:18.806750    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f1341e9-56a0-4e70-b02d-3c0fbac5f23a" containerName="extract"
Sep 14 09:52:18.806806 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:18.806768    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fee8294a-efe0-43c7-92f3-48f39e86306e" containerName="pull"
Sep 14 09:52:18.806806 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:18.806776    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fee8294a-efe0-43c7-92f3-48f39e86306e" containerName="pull"
Sep 14 09:52:18.806806 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:18.806792    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e848ae4-643a-4874-9e63-1a41fdf5288d" containerName="util"
Sep 14 09:52:18.806806 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:18.806800    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e848ae4-643a-4874-9e63-1a41fdf5288d" containerName="util"
Sep 14 09:52:18.807246 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:18.806809    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bc0e2ad-2b0e-4f8e-bae8-0cfc03237963" containerName="pull"
Sep 14 09:52:18.807246 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:18.806818    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bc0e2ad-2b0e-4f8e-bae8-0cfc03237963" containerName="pull"
Sep 14 09:52:18.807246 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:18.806827    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e848ae4-643a-4874-9e63-1a41fdf5288d" containerName="pull"
Sep 14 09:52:18.807246 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:18.806835    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e848ae4-643a-4874-9e63-1a41fdf5288d" containerName="pull"
Sep 14 09:52:18.807246 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:18.806847    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bc0e2ad-2b0e-4f8e-bae8-0cfc03237963" containerName="extract"
Sep 14 09:52:18.807246 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:18.806855    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bc0e2ad-2b0e-4f8e-bae8-0cfc03237963" containerName="extract"
Sep 14 09:52:18.807246 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:18.806864    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e848ae4-643a-4874-9e63-1a41fdf5288d" containerName="extract"
Sep 14 09:52:18.807246 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:18.806872    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e848ae4-643a-4874-9e63-1a41fdf5288d" containerName="extract"
Sep 14 09:52:18.807246 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:18.806883    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bc0e2ad-2b0e-4f8e-bae8-0cfc03237963" containerName="util"
Sep 14 09:52:18.807246 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:18.806892    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bc0e2ad-2b0e-4f8e-bae8-0cfc03237963" containerName="util"
Sep 14 09:52:18.807246 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:18.806911    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f1341e9-56a0-4e70-b02d-3c0fbac5f23a" containerName="util"
Sep 14 09:52:18.807246 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:18.806919    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f1341e9-56a0-4e70-b02d-3c0fbac5f23a" containerName="util"
Sep 14 09:52:18.807246 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:18.806929    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f1341e9-56a0-4e70-b02d-3c0fbac5f23a" containerName="pull"
Sep 14 09:52:18.807246 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:18.806937    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f1341e9-56a0-4e70-b02d-3c0fbac5f23a" containerName="pull"
Sep 14 09:52:18.807246 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:18.807044    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fee8294a-efe0-43c7-92f3-48f39e86306e" containerName="extract"
Sep 14 09:52:18.807246 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:18.807057    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e848ae4-643a-4874-9e63-1a41fdf5288d" containerName="extract"
Sep 14 09:52:18.807246 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:18.807068    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bc0e2ad-2b0e-4f8e-bae8-0cfc03237963" containerName="extract"
Sep 14 09:52:18.807246 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:18.807077    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f1341e9-56a0-4e70-b02d-3c0fbac5f23a" containerName="extract"
Sep 14 09:52:18.810510 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:18.810481    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-zqkdt"
Sep 14 09:52:18.812884 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:18.812855    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 14 09:52:18.813065 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:18.812899    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 14 09:52:18.813467 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:18.813449    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-8rxj6\""
Sep 14 09:52:18.817238 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:18.817208    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-zqkdt"]
Sep 14 09:52:18.845752 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:18.845710    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:52:18.845920 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:18.845750    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc469\" (UniqueName: \"kubernetes.io/projected/cbbafee9-948f-498f-becc-fa5827b56620-kube-api-access-jc469\") pod \"authorino-7bcfd8f9cd-zqkdt\" (UID: \"cbbafee9-948f-498f-becc-fa5827b56620\") " pod="kuadrant-system/authorino-7bcfd8f9cd-zqkdt"
Sep 14 09:52:18.946195 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:18.946157    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:52:18.946375 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:18.946311    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc469\" (UniqueName: \"kubernetes.io/projected/cbbafee9-948f-498f-becc-fa5827b56620-kube-api-access-jc469\") pod \"authorino-7bcfd8f9cd-zqkdt\" (UID: \"cbbafee9-948f-498f-becc-fa5827b56620\") " pod="kuadrant-system/authorino-7bcfd8f9cd-zqkdt"
Sep 14 09:52:18.958172 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:18.958138    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-7cpnh"]
Sep 14 09:52:18.959650 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:18.959626    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc469\" (UniqueName: \"kubernetes.io/projected/cbbafee9-948f-498f-becc-fa5827b56620-kube-api-access-jc469\") pod \"authorino-7bcfd8f9cd-zqkdt\" (UID: \"cbbafee9-948f-498f-becc-fa5827b56620\") " pod="kuadrant-system/authorino-7bcfd8f9cd-zqkdt"
Sep 14 09:52:18.961640 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:18.961616    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-7cpnh"
Sep 14 09:52:18.969175 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:18.969141    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-7cpnh"]
Sep 14 09:52:19.047228 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:19.047187    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cnvk\" (UniqueName: \"kubernetes.io/projected/70d6f2fd-33f7-4572-9833-71384c51a4a0-kube-api-access-4cnvk\") pod \"authorino-59c974bd48-7cpnh\" (UID: \"70d6f2fd-33f7-4572-9833-71384c51a4a0\") " pod="kuadrant-system/authorino-59c974bd48-7cpnh"
Sep 14 09:52:19.121267 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:19.121178    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-zqkdt"
Sep 14 09:52:19.148739 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:19.148690    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cnvk\" (UniqueName: \"kubernetes.io/projected/70d6f2fd-33f7-4572-9833-71384c51a4a0-kube-api-access-4cnvk\") pod \"authorino-59c974bd48-7cpnh\" (UID: \"70d6f2fd-33f7-4572-9833-71384c51a4a0\") " pod="kuadrant-system/authorino-59c974bd48-7cpnh"
Sep 14 09:52:19.157597 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:19.157555    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cnvk\" (UniqueName: \"kubernetes.io/projected/70d6f2fd-33f7-4572-9833-71384c51a4a0-kube-api-access-4cnvk\") pod \"authorino-59c974bd48-7cpnh\" (UID: \"70d6f2fd-33f7-4572-9833-71384c51a4a0\") " pod="kuadrant-system/authorino-59c974bd48-7cpnh"
Sep 14 09:52:19.276633 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:19.276593    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-7cpnh"
Sep 14 09:52:19.397527 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:19.397490    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-zqkdt"]
Sep 14 09:52:19.399203 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:52:19.399170    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcbbafee9_948f_498f_becc_fa5827b56620.slice/crio-da0297528b9c462a61acd27c38af469738b0765d39b49003124eda16ecd497aa WatchSource:0}: Error finding container da0297528b9c462a61acd27c38af469738b0765d39b49003124eda16ecd497aa: Status 404 returned error can't find the container with id da0297528b9c462a61acd27c38af469738b0765d39b49003124eda16ecd497aa
Sep 14 09:52:19.552651 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:19.552622    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-7cpnh"]
Sep 14 09:52:19.554984 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:52:19.554960    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod70d6f2fd_33f7_4572_9833_71384c51a4a0.slice/crio-5010df43755cc9494eac29c45de8f9e5e250ee47253b059a29680b13ae78232b WatchSource:0}: Error finding container 5010df43755cc9494eac29c45de8f9e5e250ee47253b059a29680b13ae78232b: Status 404 returned error can't find the container with id 5010df43755cc9494eac29c45de8f9e5e250ee47253b059a29680b13ae78232b
Sep 14 09:52:20.107189 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:20.107149    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-7cpnh" event={"ID":"70d6f2fd-33f7-4572-9833-71384c51a4a0","Type":"ContainerStarted","Data":"5010df43755cc9494eac29c45de8f9e5e250ee47253b059a29680b13ae78232b"}
Sep 14 09:52:20.108473 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:20.108448    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-zqkdt" event={"ID":"cbbafee9-948f-498f-becc-fa5827b56620","Type":"ContainerStarted","Data":"da0297528b9c462a61acd27c38af469738b0765d39b49003124eda16ecd497aa"}
Sep 14 09:52:23.130126 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:23.130081    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-7cpnh" event={"ID":"70d6f2fd-33f7-4572-9833-71384c51a4a0","Type":"ContainerStarted","Data":"6512f42779b114b9314f0a12f310f937c505efdf67b94e39913921701053bf7f"}
Sep 14 09:52:23.131797 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:23.131768    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-zqkdt" event={"ID":"cbbafee9-948f-498f-becc-fa5827b56620","Type":"ContainerStarted","Data":"2e860180c86e363c1b26b2d6c6219cce12a323b57acc3ee3c47de3030480466d"}
Sep 14 09:52:23.147262 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:23.147203    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-7cpnh" podStartSLOduration=2.2834559580000002 podStartE2EDuration="5.147183535s" totalImagesPullingTime="2.863727577s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:52:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:52:23.145947246 +0000 UTC m=+532.185621866" watchObservedRunningTime="2026-09-14 09:52:23.147183535 +0000 UTC m=+532.186858156"
Sep 14 09:52:23.160624 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:23.160549    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-zqkdt" podStartSLOduration=2.152771739 podStartE2EDuration="5.160525544s" totalImagesPullingTime="3.007753805s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:52:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:52:23.160306999 +0000 UTC m=+532.199981621" watchObservedRunningTime="2026-09-14 09:52:23.160525544 +0000 UTC m=+532.200200198"
Sep 14 09:52:23.201377 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:23.201332    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-zqkdt"]
Sep 14 09:52:25.142590 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:25.142535    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-zqkdt" podUID="cbbafee9-948f-498f-becc-fa5827b56620" containerName="authorino" containerID="cri-o://2e860180c86e363c1b26b2d6c6219cce12a323b57acc3ee3c47de3030480466d" gracePeriod=30
Sep 14 09:52:25.403799 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:25.403715    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-zqkdt"
Sep 14 09:52:25.508396 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:25.508363    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jc469\" (UniqueName: \"kubernetes.io/projected/cbbafee9-948f-498f-becc-fa5827b56620-kube-api-access-jc469\") pod \"cbbafee9-948f-498f-becc-fa5827b56620\" (UID: \"cbbafee9-948f-498f-becc-fa5827b56620\") "
Sep 14 09:52:25.510716 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:25.510678    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbbafee9-948f-498f-becc-fa5827b56620-kube-api-access-jc469" (OuterVolumeSpecName: "kube-api-access-jc469") pod "cbbafee9-948f-498f-becc-fa5827b56620" (UID: "cbbafee9-948f-498f-becc-fa5827b56620"). InnerVolumeSpecName "kube-api-access-jc469". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 09:52:25.609274 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:25.609218    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jc469\" (UniqueName: \"kubernetes.io/projected/cbbafee9-948f-498f-becc-fa5827b56620-kube-api-access-jc469\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:52:26.149584 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:26.149547    2563 generic.go:356] "Generic (PLEG): container finished" podID="cbbafee9-948f-498f-becc-fa5827b56620" containerID="2e860180c86e363c1b26b2d6c6219cce12a323b57acc3ee3c47de3030480466d" exitCode=0
Sep 14 09:52:26.150025 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:26.149595    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-zqkdt"
Sep 14 09:52:26.150025 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:26.149627    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-zqkdt" event={"ID":"cbbafee9-948f-498f-becc-fa5827b56620","Type":"ContainerDied","Data":"2e860180c86e363c1b26b2d6c6219cce12a323b57acc3ee3c47de3030480466d"}
Sep 14 09:52:26.150025 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:26.149673    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-zqkdt" event={"ID":"cbbafee9-948f-498f-becc-fa5827b56620","Type":"ContainerDied","Data":"da0297528b9c462a61acd27c38af469738b0765d39b49003124eda16ecd497aa"}
Sep 14 09:52:26.150025 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:26.149690    2563 scope.go:117] "RemoveContainer" containerID="2e860180c86e363c1b26b2d6c6219cce12a323b57acc3ee3c47de3030480466d"
Sep 14 09:52:26.159741 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:26.159723    2563 scope.go:117] "RemoveContainer" containerID="2e860180c86e363c1b26b2d6c6219cce12a323b57acc3ee3c47de3030480466d"
Sep 14 09:52:26.160134 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:52:26.160114    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e860180c86e363c1b26b2d6c6219cce12a323b57acc3ee3c47de3030480466d\": container with ID starting with 2e860180c86e363c1b26b2d6c6219cce12a323b57acc3ee3c47de3030480466d not found: ID does not exist" containerID="2e860180c86e363c1b26b2d6c6219cce12a323b57acc3ee3c47de3030480466d"
Sep 14 09:52:26.160211 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:26.160148    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e860180c86e363c1b26b2d6c6219cce12a323b57acc3ee3c47de3030480466d"} err="failed to get container status \"2e860180c86e363c1b26b2d6c6219cce12a323b57acc3ee3c47de3030480466d\": rpc error: code = NotFound desc = could not find container \"2e860180c86e363c1b26b2d6c6219cce12a323b57acc3ee3c47de3030480466d\": container with ID starting with 2e860180c86e363c1b26b2d6c6219cce12a323b57acc3ee3c47de3030480466d not found: ID does not exist"
Sep 14 09:52:26.174804 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:26.174769    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-zqkdt"]
Sep 14 09:52:26.177051 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:26.177018    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-zqkdt"]
Sep 14 09:52:27.486805 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:27.486766    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbbafee9-948f-498f-becc-fa5827b56620" path="/var/lib/kubelet/pods/cbbafee9-948f-498f-becc-fa5827b56620/volumes"
Sep 14 09:52:34.072378 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:34.072330    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:52:34.241658 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:34.241616    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:52:34.552367 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:34.552335    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:52:46.712884 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:46.712358    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:52:46.890483 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:46.890429    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:52:46.930178 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:46.930141    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-zdn6l"]
Sep 14 09:52:46.930549 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:46.930534    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbbafee9-948f-498f-becc-fa5827b56620" containerName="authorino"
Sep 14 09:52:46.930628 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:46.930551    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbbafee9-948f-498f-becc-fa5827b56620" containerName="authorino"
Sep 14 09:52:46.930628 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:46.930620    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbbafee9-948f-498f-becc-fa5827b56620" containerName="authorino"
Sep 14 09:52:46.937098 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:46.937066    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-zdn6l"
Sep 14 09:52:46.940680 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:46.940645    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-zdn6l"]
Sep 14 09:52:46.996873 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:46.996835    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97bjs\" (UniqueName: \"kubernetes.io/projected/be677c45-b9a5-4517-84bc-e26ca99904eb-kube-api-access-97bjs\") pod \"authorino-557f876f69-zdn6l\" (UID: \"be677c45-b9a5-4517-84bc-e26ca99904eb\") " pod="kuadrant-system/authorino-557f876f69-zdn6l"
Sep 14 09:52:47.094424 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:47.094379    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-zdn6l"]
Sep 14 09:52:47.094679 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:52:47.094659    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-97bjs], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-zdn6l" podUID="be677c45-b9a5-4517-84bc-e26ca99904eb"
Sep 14 09:52:47.098342 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:47.098309    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97bjs\" (UniqueName: \"kubernetes.io/projected/be677c45-b9a5-4517-84bc-e26ca99904eb-kube-api-access-97bjs\") pod \"authorino-557f876f69-zdn6l\" (UID: \"be677c45-b9a5-4517-84bc-e26ca99904eb\") " pod="kuadrant-system/authorino-557f876f69-zdn6l"
Sep 14 09:52:47.112670 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:47.112622    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97bjs\" (UniqueName: \"kubernetes.io/projected/be677c45-b9a5-4517-84bc-e26ca99904eb-kube-api-access-97bjs\") pod \"authorino-557f876f69-zdn6l\" (UID: \"be677c45-b9a5-4517-84bc-e26ca99904eb\") " pod="kuadrant-system/authorino-557f876f69-zdn6l"
Sep 14 09:52:47.118952 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:47.118907    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-6v2h4"]
Sep 14 09:52:47.123309 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:47.123277    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-6v2h4"
Sep 14 09:52:47.125304 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:47.125281    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 14 09:52:47.131836 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:47.131803    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-6v2h4"]
Sep 14 09:52:47.191066 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:47.191021    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-6v2h4"]
Sep 14 09:52:47.191313 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:52:47.191293    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-z5szc tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-6v2h4" podUID="36514513-dd6b-4e5e-84cf-7e2be9c2efa6"
Sep 14 09:52:47.199032 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:47.198979    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/36514513-dd6b-4e5e-84cf-7e2be9c2efa6-tls-cert\") pod \"authorino-6d4dfc649f-6v2h4\" (UID: \"36514513-dd6b-4e5e-84cf-7e2be9c2efa6\") " pod="kuadrant-system/authorino-6d4dfc649f-6v2h4"
Sep 14 09:52:47.199202 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:47.199078    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5szc\" (UniqueName: \"kubernetes.io/projected/36514513-dd6b-4e5e-84cf-7e2be9c2efa6-kube-api-access-z5szc\") pod \"authorino-6d4dfc649f-6v2h4\" (UID: \"36514513-dd6b-4e5e-84cf-7e2be9c2efa6\") " pod="kuadrant-system/authorino-6d4dfc649f-6v2h4"
Sep 14 09:52:47.225459 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:47.225417    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-fcd76d747-8rqhf"]
Sep 14 09:52:47.229304 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:47.229279    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-fcd76d747-8rqhf"
Sep 14 09:52:47.237056 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:47.237029    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-fcd76d747-8rqhf"]
Sep 14 09:52:47.259406 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:47.259320    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-zdn6l"
Sep 14 09:52:47.259575 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:47.259316    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-6v2h4"
Sep 14 09:52:47.267195 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:47.267169    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-zdn6l"
Sep 14 09:52:47.270713 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:47.270687    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-6v2h4"
Sep 14 09:52:47.299567 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:47.299521    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5szc\" (UniqueName: \"kubernetes.io/projected/36514513-dd6b-4e5e-84cf-7e2be9c2efa6-kube-api-access-z5szc\") pod \"authorino-6d4dfc649f-6v2h4\" (UID: \"36514513-dd6b-4e5e-84cf-7e2be9c2efa6\") " pod="kuadrant-system/authorino-6d4dfc649f-6v2h4"
Sep 14 09:52:47.299567 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:47.299567    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmxtm\" (UniqueName: \"kubernetes.io/projected/63eb57f4-cd2a-4769-a700-f4e14ca4c328-kube-api-access-rmxtm\") pod \"authorino-fcd76d747-8rqhf\" (UID: \"63eb57f4-cd2a-4769-a700-f4e14ca4c328\") " pod="kuadrant-system/authorino-fcd76d747-8rqhf"
Sep 14 09:52:47.299852 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:47.299611    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/63eb57f4-cd2a-4769-a700-f4e14ca4c328-tls-cert\") pod \"authorino-fcd76d747-8rqhf\" (UID: \"63eb57f4-cd2a-4769-a700-f4e14ca4c328\") " pod="kuadrant-system/authorino-fcd76d747-8rqhf"
Sep 14 09:52:47.299852 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:47.299714    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/36514513-dd6b-4e5e-84cf-7e2be9c2efa6-tls-cert\") pod \"authorino-6d4dfc649f-6v2h4\" (UID: \"36514513-dd6b-4e5e-84cf-7e2be9c2efa6\") " pod="kuadrant-system/authorino-6d4dfc649f-6v2h4"
Sep 14 09:52:47.302261 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:47.302225    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/36514513-dd6b-4e5e-84cf-7e2be9c2efa6-tls-cert\") pod \"authorino-6d4dfc649f-6v2h4\" (UID: \"36514513-dd6b-4e5e-84cf-7e2be9c2efa6\") " pod="kuadrant-system/authorino-6d4dfc649f-6v2h4"
Sep 14 09:52:47.308325 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:47.308276    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5szc\" (UniqueName: \"kubernetes.io/projected/36514513-dd6b-4e5e-84cf-7e2be9c2efa6-kube-api-access-z5szc\") pod \"authorino-6d4dfc649f-6v2h4\" (UID: \"36514513-dd6b-4e5e-84cf-7e2be9c2efa6\") " pod="kuadrant-system/authorino-6d4dfc649f-6v2h4"
Sep 14 09:52:47.400565 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:47.400523    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/36514513-dd6b-4e5e-84cf-7e2be9c2efa6-tls-cert\") pod \"36514513-dd6b-4e5e-84cf-7e2be9c2efa6\" (UID: \"36514513-dd6b-4e5e-84cf-7e2be9c2efa6\") "
Sep 14 09:52:47.400745 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:47.400580    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z5szc\" (UniqueName: \"kubernetes.io/projected/36514513-dd6b-4e5e-84cf-7e2be9c2efa6-kube-api-access-z5szc\") pod \"36514513-dd6b-4e5e-84cf-7e2be9c2efa6\" (UID: \"36514513-dd6b-4e5e-84cf-7e2be9c2efa6\") "
Sep 14 09:52:47.400745 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:47.400643    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97bjs\" (UniqueName: \"kubernetes.io/projected/be677c45-b9a5-4517-84bc-e26ca99904eb-kube-api-access-97bjs\") pod \"be677c45-b9a5-4517-84bc-e26ca99904eb\" (UID: \"be677c45-b9a5-4517-84bc-e26ca99904eb\") "
Sep 14 09:52:47.400745 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:47.400728    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmxtm\" (UniqueName: \"kubernetes.io/projected/63eb57f4-cd2a-4769-a700-f4e14ca4c328-kube-api-access-rmxtm\") pod \"authorino-fcd76d747-8rqhf\" (UID: \"63eb57f4-cd2a-4769-a700-f4e14ca4c328\") " pod="kuadrant-system/authorino-fcd76d747-8rqhf"
Sep 14 09:52:47.400918 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:47.400759    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/63eb57f4-cd2a-4769-a700-f4e14ca4c328-tls-cert\") pod \"authorino-fcd76d747-8rqhf\" (UID: \"63eb57f4-cd2a-4769-a700-f4e14ca4c328\") " pod="kuadrant-system/authorino-fcd76d747-8rqhf"
Sep 14 09:52:47.402987 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:47.402947    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be677c45-b9a5-4517-84bc-e26ca99904eb-kube-api-access-97bjs" (OuterVolumeSpecName: "kube-api-access-97bjs") pod "be677c45-b9a5-4517-84bc-e26ca99904eb" (UID: "be677c45-b9a5-4517-84bc-e26ca99904eb"). InnerVolumeSpecName "kube-api-access-97bjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 09:52:47.403141 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:47.402987    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36514513-dd6b-4e5e-84cf-7e2be9c2efa6-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "36514513-dd6b-4e5e-84cf-7e2be9c2efa6" (UID: "36514513-dd6b-4e5e-84cf-7e2be9c2efa6"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 09:52:47.403141 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:47.403041    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36514513-dd6b-4e5e-84cf-7e2be9c2efa6-kube-api-access-z5szc" (OuterVolumeSpecName: "kube-api-access-z5szc") pod "36514513-dd6b-4e5e-84cf-7e2be9c2efa6" (UID: "36514513-dd6b-4e5e-84cf-7e2be9c2efa6"). InnerVolumeSpecName "kube-api-access-z5szc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 09:52:47.403246 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:47.403226    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/63eb57f4-cd2a-4769-a700-f4e14ca4c328-tls-cert\") pod \"authorino-fcd76d747-8rqhf\" (UID: \"63eb57f4-cd2a-4769-a700-f4e14ca4c328\") " pod="kuadrant-system/authorino-fcd76d747-8rqhf"
Sep 14 09:52:47.410176 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:47.410142    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmxtm\" (UniqueName: \"kubernetes.io/projected/63eb57f4-cd2a-4769-a700-f4e14ca4c328-kube-api-access-rmxtm\") pod \"authorino-fcd76d747-8rqhf\" (UID: \"63eb57f4-cd2a-4769-a700-f4e14ca4c328\") " pod="kuadrant-system/authorino-fcd76d747-8rqhf"
Sep 14 09:52:47.502067 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:47.502027    2563 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/36514513-dd6b-4e5e-84cf-7e2be9c2efa6-tls-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:52:47.502067 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:47.502062    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5szc\" (UniqueName: \"kubernetes.io/projected/36514513-dd6b-4e5e-84cf-7e2be9c2efa6-kube-api-access-z5szc\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:52:47.502067 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:47.502071    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97bjs\" (UniqueName: \"kubernetes.io/projected/be677c45-b9a5-4517-84bc-e26ca99904eb-kube-api-access-97bjs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:52:47.540081 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:47.539923    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-fcd76d747-8rqhf"
Sep 14 09:52:47.817260 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:47.817230    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-fcd76d747-8rqhf"]
Sep 14 09:52:47.819243 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:52:47.819205    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod63eb57f4_cd2a_4769_a700_f4e14ca4c328.slice/crio-195e0bcfd872cee7f6ece41e81491a017fe2ca3380967cd608473d122b1813e3 WatchSource:0}: Error finding container 195e0bcfd872cee7f6ece41e81491a017fe2ca3380967cd608473d122b1813e3: Status 404 returned error can't find the container with id 195e0bcfd872cee7f6ece41e81491a017fe2ca3380967cd608473d122b1813e3
Sep 14 09:52:48.011631 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:48.011583    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:52:48.081053 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:48.080945    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:52:48.266297 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:48.266257    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-fcd76d747-8rqhf" event={"ID":"63eb57f4-cd2a-4769-a700-f4e14ca4c328","Type":"ContainerStarted","Data":"195e0bcfd872cee7f6ece41e81491a017fe2ca3380967cd608473d122b1813e3"}
Sep 14 09:52:48.266297 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:48.266300    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-6v2h4"
Sep 14 09:52:48.266572 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:48.266300    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-zdn6l"
Sep 14 09:52:48.301142 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:48.301092    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-zdn6l"]
Sep 14 09:52:48.306222 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:48.306186    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-zdn6l"]
Sep 14 09:52:48.331249 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:48.331142    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-6v2h4"]
Sep 14 09:52:48.333374 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:48.333340    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-6v2h4"]
Sep 14 09:52:49.273885 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:49.273848    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-fcd76d747-8rqhf" event={"ID":"63eb57f4-cd2a-4769-a700-f4e14ca4c328","Type":"ContainerStarted","Data":"cc1750999350c4f0b616ef6d0f3db02829023c358a776af8a71d2ba2001ba199"}
Sep 14 09:52:49.292798 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:49.292721    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-fcd76d747-8rqhf" podStartSLOduration=1.725742512 podStartE2EDuration="2.292696396s" totalImagesPullingTime="566.953884ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:52:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:52:49.291845136 +0000 UTC m=+558.331519782" watchObservedRunningTime="2026-09-14 09:52:49.292696396 +0000 UTC m=+558.332371019"
Sep 14 09:52:49.321444 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:49.321399    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-7cpnh"]
Sep 14 09:52:49.321659 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:49.321630    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-7cpnh" podUID="70d6f2fd-33f7-4572-9833-71384c51a4a0" containerName="authorino" containerID="cri-o://6512f42779b114b9314f0a12f310f937c505efdf67b94e39913921701053bf7f" gracePeriod=30
Sep 14 09:52:49.488164 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:49.488121    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36514513-dd6b-4e5e-84cf-7e2be9c2efa6" path="/var/lib/kubelet/pods/36514513-dd6b-4e5e-84cf-7e2be9c2efa6/volumes"
Sep 14 09:52:49.488440 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:49.488425    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be677c45-b9a5-4517-84bc-e26ca99904eb" path="/var/lib/kubelet/pods/be677c45-b9a5-4517-84bc-e26ca99904eb/volumes"
Sep 14 09:52:49.581305 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:49.581273    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-7cpnh"
Sep 14 09:52:49.723964 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:49.723914    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4cnvk\" (UniqueName: \"kubernetes.io/projected/70d6f2fd-33f7-4572-9833-71384c51a4a0-kube-api-access-4cnvk\") pod \"70d6f2fd-33f7-4572-9833-71384c51a4a0\" (UID: \"70d6f2fd-33f7-4572-9833-71384c51a4a0\") "
Sep 14 09:52:49.726354 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:49.726303    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70d6f2fd-33f7-4572-9833-71384c51a4a0-kube-api-access-4cnvk" (OuterVolumeSpecName: "kube-api-access-4cnvk") pod "70d6f2fd-33f7-4572-9833-71384c51a4a0" (UID: "70d6f2fd-33f7-4572-9833-71384c51a4a0"). InnerVolumeSpecName "kube-api-access-4cnvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 09:52:49.825417 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:49.825323    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cnvk\" (UniqueName: \"kubernetes.io/projected/70d6f2fd-33f7-4572-9833-71384c51a4a0-kube-api-access-4cnvk\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:52:50.281409 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:50.281372    2563 generic.go:356] "Generic (PLEG): container finished" podID="70d6f2fd-33f7-4572-9833-71384c51a4a0" containerID="6512f42779b114b9314f0a12f310f937c505efdf67b94e39913921701053bf7f" exitCode=0
Sep 14 09:52:50.281819 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:50.281422    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-7cpnh"
Sep 14 09:52:50.281819 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:50.281461    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-7cpnh" event={"ID":"70d6f2fd-33f7-4572-9833-71384c51a4a0","Type":"ContainerDied","Data":"6512f42779b114b9314f0a12f310f937c505efdf67b94e39913921701053bf7f"}
Sep 14 09:52:50.281819 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:50.281502    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-7cpnh" event={"ID":"70d6f2fd-33f7-4572-9833-71384c51a4a0","Type":"ContainerDied","Data":"5010df43755cc9494eac29c45de8f9e5e250ee47253b059a29680b13ae78232b"}
Sep 14 09:52:50.281819 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:50.281522    2563 scope.go:117] "RemoveContainer" containerID="6512f42779b114b9314f0a12f310f937c505efdf67b94e39913921701053bf7f"
Sep 14 09:52:50.290920 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:50.290897    2563 scope.go:117] "RemoveContainer" containerID="6512f42779b114b9314f0a12f310f937c505efdf67b94e39913921701053bf7f"
Sep 14 09:52:50.291330 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:52:50.291307    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6512f42779b114b9314f0a12f310f937c505efdf67b94e39913921701053bf7f\": container with ID starting with 6512f42779b114b9314f0a12f310f937c505efdf67b94e39913921701053bf7f not found: ID does not exist" containerID="6512f42779b114b9314f0a12f310f937c505efdf67b94e39913921701053bf7f"
Sep 14 09:52:50.291405 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:50.291340    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6512f42779b114b9314f0a12f310f937c505efdf67b94e39913921701053bf7f"} err="failed to get container status \"6512f42779b114b9314f0a12f310f937c505efdf67b94e39913921701053bf7f\": rpc error: code = NotFound desc = could not find container \"6512f42779b114b9314f0a12f310f937c505efdf67b94e39913921701053bf7f\": container with ID starting with 6512f42779b114b9314f0a12f310f937c505efdf67b94e39913921701053bf7f not found: ID does not exist"
Sep 14 09:52:50.310396 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:50.310351    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-7cpnh"]
Sep 14 09:52:50.315693 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:50.315649    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-7cpnh"]
Sep 14 09:52:51.485376 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:51.485319    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70d6f2fd-33f7-4572-9833-71384c51a4a0" path="/var/lib/kubelet/pods/70d6f2fd-33f7-4572-9833-71384c51a4a0/volumes"
Sep 14 09:52:53.968088 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:53.968036    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:52:54.069357 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:54.069306    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:52:54.169305 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:54.169269    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:52:54.691700 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:54.691662    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:52:58.180983 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:58.180939    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:52:58.277153 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:52:58.277104    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:53:15.883887 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:53:15.883852    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:53:15.985319 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:53:15.985279    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:53:28.343188 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:53:28.343145    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:53:28.443979 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:53:28.443935    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:53:28.900138 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:53:28.900087    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:53:29.895775 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:53:29.895736    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:53:29.996513 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:53:29.996478    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:53:30.111942 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:53:30.111907    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:53:30.213760 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:53:30.213665    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:53:30.602941 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:53:30.602900    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:53:30.682775 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:53:30.682740    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:53:40.920616 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:53:40.920579    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:53:41.017014 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:53:41.016969    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:53:51.772791 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:53:51.772753    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5bhm9"]
Sep 14 09:53:51.773343 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:53:51.773267    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70d6f2fd-33f7-4572-9833-71384c51a4a0" containerName="authorino"
Sep 14 09:53:51.773343 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:53:51.773286    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="70d6f2fd-33f7-4572-9833-71384c51a4a0" containerName="authorino"
Sep 14 09:53:51.773473 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:53:51.773385    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="70d6f2fd-33f7-4572-9833-71384c51a4a0" containerName="authorino"
Sep 14 09:53:51.776538 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:53:51.776509    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5bhm9"
Sep 14 09:53:51.779090 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:53:51.779054    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 14 09:53:51.779756 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:53:51.779733    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 14 09:53:51.779878 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:53:51.779733    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 14 09:53:51.779878 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:53:51.779734    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-8sh6p\""
Sep 14 09:53:51.786737 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:53:51.786704    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5bhm9"]
Sep 14 09:53:51.809451 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:53:51.809411    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:53:51.847671 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:53:51.847624    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a6aa52e-92c8-498f-a935-75ef0ea46e6a-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5bhm9\" (UID: \"1a6aa52e-92c8-498f-a935-75ef0ea46e6a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5bhm9"
Sep 14 09:53:51.847861 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:53:51.847681    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a6aa52e-92c8-498f-a935-75ef0ea46e6a-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5bhm9\" (UID: \"1a6aa52e-92c8-498f-a935-75ef0ea46e6a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5bhm9"
Sep 14 09:53:51.847861 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:53:51.847747    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a6aa52e-92c8-498f-a935-75ef0ea46e6a-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5bhm9\" (UID: \"1a6aa52e-92c8-498f-a935-75ef0ea46e6a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5bhm9"
Sep 14 09:53:51.847861 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:53:51.847762    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a6aa52e-92c8-498f-a935-75ef0ea46e6a-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5bhm9\" (UID: \"1a6aa52e-92c8-498f-a935-75ef0ea46e6a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5bhm9"
Sep 14 09:53:51.847861 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:53:51.847801    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjdhm\" (UniqueName: \"kubernetes.io/projected/1a6aa52e-92c8-498f-a935-75ef0ea46e6a-kube-api-access-fjdhm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5bhm9\" (UID: \"1a6aa52e-92c8-498f-a935-75ef0ea46e6a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5bhm9"
Sep 14 09:53:51.847861 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:53:51.847821    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a6aa52e-92c8-498f-a935-75ef0ea46e6a-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5bhm9\" (UID: \"1a6aa52e-92c8-498f-a935-75ef0ea46e6a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5bhm9"
Sep 14 09:53:51.948787 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:53:51.948746    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a6aa52e-92c8-498f-a935-75ef0ea46e6a-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5bhm9\" (UID: \"1a6aa52e-92c8-498f-a935-75ef0ea46e6a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5bhm9"
Sep 14 09:53:51.948787 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:53:51.948787    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a6aa52e-92c8-498f-a935-75ef0ea46e6a-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5bhm9\" (UID: \"1a6aa52e-92c8-498f-a935-75ef0ea46e6a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5bhm9"
Sep 14 09:53:51.949064 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:53:51.948911    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjdhm\" (UniqueName: \"kubernetes.io/projected/1a6aa52e-92c8-498f-a935-75ef0ea46e6a-kube-api-access-fjdhm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5bhm9\" (UID: \"1a6aa52e-92c8-498f-a935-75ef0ea46e6a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5bhm9"
Sep 14 09:53:51.949064 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:53:51.948953    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a6aa52e-92c8-498f-a935-75ef0ea46e6a-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5bhm9\" (UID: \"1a6aa52e-92c8-498f-a935-75ef0ea46e6a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5bhm9"
Sep 14 09:53:51.949183 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:53:51.949065    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a6aa52e-92c8-498f-a935-75ef0ea46e6a-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5bhm9\" (UID: \"1a6aa52e-92c8-498f-a935-75ef0ea46e6a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5bhm9"
Sep 14 09:53:51.949183 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:53:51.949103    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a6aa52e-92c8-498f-a935-75ef0ea46e6a-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5bhm9\" (UID: \"1a6aa52e-92c8-498f-a935-75ef0ea46e6a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5bhm9"
Sep 14 09:53:51.949183 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:53:51.949155    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a6aa52e-92c8-498f-a935-75ef0ea46e6a-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5bhm9\" (UID: \"1a6aa52e-92c8-498f-a935-75ef0ea46e6a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5bhm9"
Sep 14 09:53:51.949332 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:53:51.949213    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a6aa52e-92c8-498f-a935-75ef0ea46e6a-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5bhm9\" (UID: \"1a6aa52e-92c8-498f-a935-75ef0ea46e6a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5bhm9"
Sep 14 09:53:51.949332 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:53:51.949294    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a6aa52e-92c8-498f-a935-75ef0ea46e6a-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5bhm9\" (UID: \"1a6aa52e-92c8-498f-a935-75ef0ea46e6a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5bhm9"
Sep 14 09:53:51.951498 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:53:51.951463    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a6aa52e-92c8-498f-a935-75ef0ea46e6a-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5bhm9\" (UID: \"1a6aa52e-92c8-498f-a935-75ef0ea46e6a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5bhm9"
Sep 14 09:53:51.951680 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:53:51.951661    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a6aa52e-92c8-498f-a935-75ef0ea46e6a-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5bhm9\" (UID: \"1a6aa52e-92c8-498f-a935-75ef0ea46e6a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5bhm9"
Sep 14 09:53:51.957882 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:53:51.957850    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjdhm\" (UniqueName: \"kubernetes.io/projected/1a6aa52e-92c8-498f-a935-75ef0ea46e6a-kube-api-access-fjdhm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5bhm9\" (UID: \"1a6aa52e-92c8-498f-a935-75ef0ea46e6a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5bhm9"
Sep 14 09:53:52.087957 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:53:52.087856    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5bhm9"
Sep 14 09:53:52.403815 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:53:52.403786    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5bhm9"]
Sep 14 09:53:52.407289 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:53:52.407251    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a6aa52e_92c8_498f_a935_75ef0ea46e6a.slice/crio-aafa7ea536b6cc93aa893248662354ef73b5ff83972ac13c820ad64af1e87bec WatchSource:0}: Error finding container aafa7ea536b6cc93aa893248662354ef73b5ff83972ac13c820ad64af1e87bec: Status 404 returned error can't find the container with id aafa7ea536b6cc93aa893248662354ef73b5ff83972ac13c820ad64af1e87bec
Sep 14 09:53:52.409077 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:53:52.409050    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 14 09:53:52.617698 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:53:52.617649    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5bhm9" event={"ID":"1a6aa52e-92c8-498f-a935-75ef0ea46e6a","Type":"ContainerStarted","Data":"aafa7ea536b6cc93aa893248662354ef73b5ff83972ac13c820ad64af1e87bec"}
Sep 14 09:53:53.627572 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:53:53.627535    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5bhm9" event={"ID":"1a6aa52e-92c8-498f-a935-75ef0ea46e6a","Type":"ContainerStarted","Data":"e02973d12bf8a87e6cfc8c33c9d086a3217f7e746f9efaba12080a8a842f2578"}
Sep 14 09:53:53.676255 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:53:53.676186    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5bhm9" podStartSLOduration=1.604215784 podStartE2EDuration="2.676168896s" totalImagesPullingTime="1.071953112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:53:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:53:53.675331469 +0000 UTC m=+622.715006092" watchObservedRunningTime="2026-09-14 09:53:53.676168896 +0000 UTC m=+622.715843516"
Sep 14 09:53:53.806467 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:53:53.806424    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:54:02.089106 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:54:02.089062    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5bhm9"
Sep 14 09:54:02.089620 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:54:02.089130    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5bhm9"
Sep 14 09:54:02.101724 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:54:02.101693    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5bhm9"
Sep 14 09:54:02.683282 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:54:02.683250    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5bhm9"
Sep 14 09:54:04.179731 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:54:04.179684    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:54:05.513529 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:54:05.513485    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:54:15.262168 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:54:15.262129    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-7vg96"]
Sep 14 09:54:15.265621 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:54:15.265596    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7vg96"
Sep 14 09:54:15.267925 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:54:15.267903    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 14 09:54:15.278782 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:54:15.278744    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-7vg96"]
Sep 14 09:54:15.350847 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:54:15.350813    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:54:15.359507 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:54:15.359459    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2f5a76a-2f1f-44b6-877b-943f7a78e4b8-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7vg96\" (UID: \"c2f5a76a-2f1f-44b6-877b-943f7a78e4b8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7vg96"
Sep 14 09:54:15.359658 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:54:15.359525    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2f5a76a-2f1f-44b6-877b-943f7a78e4b8-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7vg96\" (UID: \"c2f5a76a-2f1f-44b6-877b-943f7a78e4b8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7vg96"
Sep 14 09:54:15.359658 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:54:15.359568    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr7v9\" (UniqueName: \"kubernetes.io/projected/c2f5a76a-2f1f-44b6-877b-943f7a78e4b8-kube-api-access-xr7v9\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7vg96\" (UID: \"c2f5a76a-2f1f-44b6-877b-943f7a78e4b8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7vg96"
Sep 14 09:54:15.359658 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:54:15.359589    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2f5a76a-2f1f-44b6-877b-943f7a78e4b8-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7vg96\" (UID: \"c2f5a76a-2f1f-44b6-877b-943f7a78e4b8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7vg96"
Sep 14 09:54:15.359658 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:54:15.359646    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2f5a76a-2f1f-44b6-877b-943f7a78e4b8-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7vg96\" (UID: \"c2f5a76a-2f1f-44b6-877b-943f7a78e4b8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7vg96"
Sep 14 09:54:15.359808 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:54:15.359670    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2f5a76a-2f1f-44b6-877b-943f7a78e4b8-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7vg96\" (UID: \"c2f5a76a-2f1f-44b6-877b-943f7a78e4b8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7vg96"
Sep 14 09:54:15.460785 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:54:15.460741    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2f5a76a-2f1f-44b6-877b-943f7a78e4b8-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7vg96\" (UID: \"c2f5a76a-2f1f-44b6-877b-943f7a78e4b8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7vg96"
Sep 14 09:54:15.461084 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:54:15.460797    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2f5a76a-2f1f-44b6-877b-943f7a78e4b8-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7vg96\" (UID: \"c2f5a76a-2f1f-44b6-877b-943f7a78e4b8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7vg96"
Sep 14 09:54:15.461084 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:54:15.460833    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2f5a76a-2f1f-44b6-877b-943f7a78e4b8-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7vg96\" (UID: \"c2f5a76a-2f1f-44b6-877b-943f7a78e4b8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7vg96"
Sep 14 09:54:15.461084 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:54:15.460855    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr7v9\" (UniqueName: \"kubernetes.io/projected/c2f5a76a-2f1f-44b6-877b-943f7a78e4b8-kube-api-access-xr7v9\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7vg96\" (UID: \"c2f5a76a-2f1f-44b6-877b-943f7a78e4b8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7vg96"
Sep 14 09:54:15.461084 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:54:15.460875    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2f5a76a-2f1f-44b6-877b-943f7a78e4b8-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7vg96\" (UID: \"c2f5a76a-2f1f-44b6-877b-943f7a78e4b8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7vg96"
Sep 14 09:54:15.461084 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:54:15.460916    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2f5a76a-2f1f-44b6-877b-943f7a78e4b8-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7vg96\" (UID: \"c2f5a76a-2f1f-44b6-877b-943f7a78e4b8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7vg96"
Sep 14 09:54:15.461368 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:54:15.461312    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2f5a76a-2f1f-44b6-877b-943f7a78e4b8-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7vg96\" (UID: \"c2f5a76a-2f1f-44b6-877b-943f7a78e4b8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7vg96"
Sep 14 09:54:15.461368 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:54:15.461348    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2f5a76a-2f1f-44b6-877b-943f7a78e4b8-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7vg96\" (UID: \"c2f5a76a-2f1f-44b6-877b-943f7a78e4b8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7vg96"
Sep 14 09:54:15.461446 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:54:15.461400    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2f5a76a-2f1f-44b6-877b-943f7a78e4b8-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7vg96\" (UID: \"c2f5a76a-2f1f-44b6-877b-943f7a78e4b8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7vg96"
Sep 14 09:54:15.463330 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:54:15.463304    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2f5a76a-2f1f-44b6-877b-943f7a78e4b8-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7vg96\" (UID: \"c2f5a76a-2f1f-44b6-877b-943f7a78e4b8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7vg96"
Sep 14 09:54:15.463516 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:54:15.463496    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2f5a76a-2f1f-44b6-877b-943f7a78e4b8-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7vg96\" (UID: \"c2f5a76a-2f1f-44b6-877b-943f7a78e4b8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7vg96"
Sep 14 09:54:15.468972 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:54:15.468936    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr7v9\" (UniqueName: \"kubernetes.io/projected/c2f5a76a-2f1f-44b6-877b-943f7a78e4b8-kube-api-access-xr7v9\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7vg96\" (UID: \"c2f5a76a-2f1f-44b6-877b-943f7a78e4b8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7vg96"
Sep 14 09:54:15.576883 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:54:15.576790    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7vg96"
Sep 14 09:54:15.855015 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:54:15.854964    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-7vg96"]
Sep 14 09:54:15.857041 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:54:15.856986    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2f5a76a_2f1f_44b6_877b_943f7a78e4b8.slice/crio-e2f988c537503da629ef924f60982b196beba0b209ed80397e9357a9dea752de WatchSource:0}: Error finding container e2f988c537503da629ef924f60982b196beba0b209ed80397e9357a9dea752de: Status 404 returned error can't find the container with id e2f988c537503da629ef924f60982b196beba0b209ed80397e9357a9dea752de
Sep 14 09:54:16.748195 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:54:16.748153    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7vg96" event={"ID":"c2f5a76a-2f1f-44b6-877b-943f7a78e4b8","Type":"ContainerStarted","Data":"9f509b7b5dbb60b2d3921f96f974828067d5e01fc51532de023e2d08253ba315"}
Sep 14 09:54:16.748596 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:54:16.748217    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7vg96" event={"ID":"c2f5a76a-2f1f-44b6-877b-943f7a78e4b8","Type":"ContainerStarted","Data":"e2f988c537503da629ef924f60982b196beba0b209ed80397e9357a9dea752de"}
Sep 14 09:54:16.770987 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:54:16.770921    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7vg96" podStartSLOduration=1.56783291 podStartE2EDuration="1.770901019s" totalImagesPullingTime="203.068109ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:54:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:54:16.767755797 +0000 UTC m=+645.807430412" watchObservedRunningTime="2026-09-14 09:54:16.770901019 +0000 UTC m=+645.810575680"
Sep 14 09:54:24.682329 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:54:24.682286    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:54:25.577915 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:54:25.577862    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7vg96"
Sep 14 09:54:25.578130 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:54:25.577943    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7vg96"
Sep 14 09:54:25.590683 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:54:25.590651    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7vg96"
Sep 14 09:54:25.808611 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:54:25.808578    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7vg96"
Sep 14 09:54:37.297045 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:54:37.296993    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:54:44.518824 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:54:44.518792    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:54:54.511304 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:54:54.511269    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:55:05.784137 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:55:05.784100    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:55:56.928522 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:55:56.928482    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:01.383699 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:01.383607    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-86d8ccc4f4-xsck7"]
Sep 14 09:57:01.387045 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:01.386993    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-86d8ccc4f4-xsck7"
Sep 14 09:57:01.412456 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:01.412423    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-86d8ccc4f4-xsck7"]
Sep 14 09:57:01.520566 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:01.520521    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8v9j\" (UniqueName: \"kubernetes.io/projected/df7eded7-4c9c-4d49-9ae0-4cb3c8eeb136-kube-api-access-z8v9j\") pod \"authorino-86d8ccc4f4-xsck7\" (UID: \"df7eded7-4c9c-4d49-9ae0-4cb3c8eeb136\") " pod="kuadrant-system/authorino-86d8ccc4f4-xsck7"
Sep 14 09:57:01.520566 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:01.520580    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/df7eded7-4c9c-4d49-9ae0-4cb3c8eeb136-tls-cert\") pod \"authorino-86d8ccc4f4-xsck7\" (UID: \"df7eded7-4c9c-4d49-9ae0-4cb3c8eeb136\") " pod="kuadrant-system/authorino-86d8ccc4f4-xsck7"
Sep 14 09:57:01.621651 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:01.621594    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/df7eded7-4c9c-4d49-9ae0-4cb3c8eeb136-tls-cert\") pod \"authorino-86d8ccc4f4-xsck7\" (UID: \"df7eded7-4c9c-4d49-9ae0-4cb3c8eeb136\") " pod="kuadrant-system/authorino-86d8ccc4f4-xsck7"
Sep 14 09:57:01.622157 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:01.622121    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8v9j\" (UniqueName: \"kubernetes.io/projected/df7eded7-4c9c-4d49-9ae0-4cb3c8eeb136-kube-api-access-z8v9j\") pod \"authorino-86d8ccc4f4-xsck7\" (UID: \"df7eded7-4c9c-4d49-9ae0-4cb3c8eeb136\") " pod="kuadrant-system/authorino-86d8ccc4f4-xsck7"
Sep 14 09:57:01.624394 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:01.624367    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/df7eded7-4c9c-4d49-9ae0-4cb3c8eeb136-tls-cert\") pod \"authorino-86d8ccc4f4-xsck7\" (UID: \"df7eded7-4c9c-4d49-9ae0-4cb3c8eeb136\") " pod="kuadrant-system/authorino-86d8ccc4f4-xsck7"
Sep 14 09:57:01.630664 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:01.630635    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8v9j\" (UniqueName: \"kubernetes.io/projected/df7eded7-4c9c-4d49-9ae0-4cb3c8eeb136-kube-api-access-z8v9j\") pod \"authorino-86d8ccc4f4-xsck7\" (UID: \"df7eded7-4c9c-4d49-9ae0-4cb3c8eeb136\") " pod="kuadrant-system/authorino-86d8ccc4f4-xsck7"
Sep 14 09:57:01.696497 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:01.696401    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-86d8ccc4f4-xsck7"
Sep 14 09:57:01.975750 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:01.975721    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-86d8ccc4f4-xsck7"]
Sep 14 09:57:01.978103 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:57:01.978076    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddf7eded7_4c9c_4d49_9ae0_4cb3c8eeb136.slice/crio-9bd23b99de61fd5426e563b7fec1c62b70bfd0f3cc729d186d7d0c13b2c03b56 WatchSource:0}: Error finding container 9bd23b99de61fd5426e563b7fec1c62b70bfd0f3cc729d186d7d0c13b2c03b56: Status 404 returned error can't find the container with id 9bd23b99de61fd5426e563b7fec1c62b70bfd0f3cc729d186d7d0c13b2c03b56
Sep 14 09:57:02.635815 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:02.635772    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-86d8ccc4f4-xsck7" event={"ID":"df7eded7-4c9c-4d49-9ae0-4cb3c8eeb136","Type":"ContainerStarted","Data":"9bd23b99de61fd5426e563b7fec1c62b70bfd0f3cc729d186d7d0c13b2c03b56"}
Sep 14 09:57:03.644106 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:03.644056    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-86d8ccc4f4-xsck7" event={"ID":"df7eded7-4c9c-4d49-9ae0-4cb3c8eeb136","Type":"ContainerStarted","Data":"930303bbe7fe8795acfb9337401bfb74409779ec56184594d5fd5184469e461b"}
Sep 14 09:57:03.666628 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:03.666554    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-86d8ccc4f4-xsck7" podStartSLOduration=1.792473661 podStartE2EDuration="2.666528907s" totalImagesPullingTime="874.055246ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:57:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:57:03.665370857 +0000 UTC m=+812.705045479" watchObservedRunningTime="2026-09-14 09:57:03.666528907 +0000 UTC m=+812.706203529"
Sep 14 09:57:03.699723 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:03.699686    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-fcd76d747-8rqhf"]
Sep 14 09:57:03.699942 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:03.699880    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-fcd76d747-8rqhf" podUID="63eb57f4-cd2a-4769-a700-f4e14ca4c328" containerName="authorino" containerID="cri-o://cc1750999350c4f0b616ef6d0f3db02829023c358a776af8a71d2ba2001ba199" gracePeriod=30
Sep 14 09:57:03.970051 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:03.970015    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-fcd76d747-8rqhf"
Sep 14 09:57:04.043108 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:04.043060    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rmxtm\" (UniqueName: \"kubernetes.io/projected/63eb57f4-cd2a-4769-a700-f4e14ca4c328-kube-api-access-rmxtm\") pod \"63eb57f4-cd2a-4769-a700-f4e14ca4c328\" (UID: \"63eb57f4-cd2a-4769-a700-f4e14ca4c328\") "
Sep 14 09:57:04.043108 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:04.043112    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/63eb57f4-cd2a-4769-a700-f4e14ca4c328-tls-cert\") pod \"63eb57f4-cd2a-4769-a700-f4e14ca4c328\" (UID: \"63eb57f4-cd2a-4769-a700-f4e14ca4c328\") "
Sep 14 09:57:04.045312 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:04.045278    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63eb57f4-cd2a-4769-a700-f4e14ca4c328-kube-api-access-rmxtm" (OuterVolumeSpecName: "kube-api-access-rmxtm") pod "63eb57f4-cd2a-4769-a700-f4e14ca4c328" (UID: "63eb57f4-cd2a-4769-a700-f4e14ca4c328"). InnerVolumeSpecName "kube-api-access-rmxtm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 09:57:04.055470 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:04.055421    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63eb57f4-cd2a-4769-a700-f4e14ca4c328-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "63eb57f4-cd2a-4769-a700-f4e14ca4c328" (UID: "63eb57f4-cd2a-4769-a700-f4e14ca4c328"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 09:57:04.144419 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:04.144376    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmxtm\" (UniqueName: \"kubernetes.io/projected/63eb57f4-cd2a-4769-a700-f4e14ca4c328-kube-api-access-rmxtm\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:57:04.144419 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:04.144410    2563 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/63eb57f4-cd2a-4769-a700-f4e14ca4c328-tls-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:57:04.245869 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:04.245827    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:04.335395 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:04.335356    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:04.653423 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:04.653329    2563 generic.go:356] "Generic (PLEG): container finished" podID="63eb57f4-cd2a-4769-a700-f4e14ca4c328" containerID="cc1750999350c4f0b616ef6d0f3db02829023c358a776af8a71d2ba2001ba199" exitCode=0
Sep 14 09:57:04.653423 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:04.653376    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-fcd76d747-8rqhf"
Sep 14 09:57:04.653423 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:04.653410    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-fcd76d747-8rqhf" event={"ID":"63eb57f4-cd2a-4769-a700-f4e14ca4c328","Type":"ContainerDied","Data":"cc1750999350c4f0b616ef6d0f3db02829023c358a776af8a71d2ba2001ba199"}
Sep 14 09:57:04.654100 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:04.653460    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-fcd76d747-8rqhf" event={"ID":"63eb57f4-cd2a-4769-a700-f4e14ca4c328","Type":"ContainerDied","Data":"195e0bcfd872cee7f6ece41e81491a017fe2ca3380967cd608473d122b1813e3"}
Sep 14 09:57:04.654100 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:04.653482    2563 scope.go:117] "RemoveContainer" containerID="cc1750999350c4f0b616ef6d0f3db02829023c358a776af8a71d2ba2001ba199"
Sep 14 09:57:04.664814 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:04.664792    2563 scope.go:117] "RemoveContainer" containerID="cc1750999350c4f0b616ef6d0f3db02829023c358a776af8a71d2ba2001ba199"
Sep 14 09:57:04.665198 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:57:04.665172    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc1750999350c4f0b616ef6d0f3db02829023c358a776af8a71d2ba2001ba199\": container with ID starting with cc1750999350c4f0b616ef6d0f3db02829023c358a776af8a71d2ba2001ba199 not found: ID does not exist" containerID="cc1750999350c4f0b616ef6d0f3db02829023c358a776af8a71d2ba2001ba199"
Sep 14 09:57:04.665257 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:04.665208    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc1750999350c4f0b616ef6d0f3db02829023c358a776af8a71d2ba2001ba199"} err="failed to get container status \"cc1750999350c4f0b616ef6d0f3db02829023c358a776af8a71d2ba2001ba199\": rpc error: code = NotFound desc = could not find container \"cc1750999350c4f0b616ef6d0f3db02829023c358a776af8a71d2ba2001ba199\": container with ID starting with cc1750999350c4f0b616ef6d0f3db02829023c358a776af8a71d2ba2001ba199 not found: ID does not exist"
Sep 14 09:57:04.685727 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:04.685673    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-fcd76d747-8rqhf"]
Sep 14 09:57:04.696149 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:04.696110    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-fcd76d747-8rqhf"]
Sep 14 09:57:05.358487 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:05.358446    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:05.440976 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:05.440917    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:05.482805 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:05.482762    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63eb57f4-cd2a-4769-a700-f4e14ca4c328" path="/var/lib/kubelet/pods/63eb57f4-cd2a-4769-a700-f4e14ca4c328/volumes"
Sep 14 09:57:20.786797 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:20.786760    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:20.899628 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:20.899591    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:21.003294 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.003259    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:21.138400 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.138148    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m"]
Sep 14 09:57:21.139334 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.139302    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63eb57f4-cd2a-4769-a700-f4e14ca4c328" containerName="authorino"
Sep 14 09:57:21.139334 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.139334    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="63eb57f4-cd2a-4769-a700-f4e14ca4c328" containerName="authorino"
Sep 14 09:57:21.139545 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.139479    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="63eb57f4-cd2a-4769-a700-f4e14ca4c328" containerName="authorino"
Sep 14 09:57:21.144412 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.144384    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m"
Sep 14 09:57:21.147707 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.147676    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-2901ec-openshift-default-dockercfg-56pxf\""
Sep 14 09:57:21.157055 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.157021    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m"]
Sep 14 09:57:21.160564 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.160530    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:21.188752 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.188717    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3188c970-5b86-4c03-95cc-12480aa11882-istio-data\") pod \"e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m\" (UID: \"3188c970-5b86-4c03-95cc-12480aa11882\") " pod="openshift-ingress/e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m"
Sep 14 09:57:21.188752 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.188761    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3188c970-5b86-4c03-95cc-12480aa11882-istio-envoy\") pod \"e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m\" (UID: \"3188c970-5b86-4c03-95cc-12480aa11882\") " pod="openshift-ingress/e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m"
Sep 14 09:57:21.188979 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.188791    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3188c970-5b86-4c03-95cc-12480aa11882-istiod-ca-cert\") pod \"e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m\" (UID: \"3188c970-5b86-4c03-95cc-12480aa11882\") " pod="openshift-ingress/e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m"
Sep 14 09:57:21.188979 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.188909    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdp8m\" (UniqueName: \"kubernetes.io/projected/3188c970-5b86-4c03-95cc-12480aa11882-kube-api-access-mdp8m\") pod \"e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m\" (UID: \"3188c970-5b86-4c03-95cc-12480aa11882\") " pod="openshift-ingress/e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m"
Sep 14 09:57:21.188979 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.188964    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3188c970-5b86-4c03-95cc-12480aa11882-istio-token\") pod \"e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m\" (UID: \"3188c970-5b86-4c03-95cc-12480aa11882\") " pod="openshift-ingress/e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m"
Sep 14 09:57:21.189121 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.189024    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3188c970-5b86-4c03-95cc-12480aa11882-istio-podinfo\") pod \"e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m\" (UID: \"3188c970-5b86-4c03-95cc-12480aa11882\") " pod="openshift-ingress/e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m"
Sep 14 09:57:21.189121 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.189066    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3188c970-5b86-4c03-95cc-12480aa11882-workload-socket\") pod \"e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m\" (UID: \"3188c970-5b86-4c03-95cc-12480aa11882\") " pod="openshift-ingress/e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m"
Sep 14 09:57:21.189121 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.189094    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3188c970-5b86-4c03-95cc-12480aa11882-workload-certs\") pod \"e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m\" (UID: \"3188c970-5b86-4c03-95cc-12480aa11882\") " pod="openshift-ingress/e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m"
Sep 14 09:57:21.189240 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.189198    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3188c970-5b86-4c03-95cc-12480aa11882-credential-socket\") pod \"e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m\" (UID: \"3188c970-5b86-4c03-95cc-12480aa11882\") " pod="openshift-ingress/e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m"
Sep 14 09:57:21.213729 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.213675    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn"]
Sep 14 09:57:21.217394 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.217365    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn"
Sep 14 09:57:21.221206 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.221173    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-0e60f3-openshift-default-dockercfg-kt648\""
Sep 14 09:57:21.234916 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.234878    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn"]
Sep 14 09:57:21.273883 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.273841    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:21.290100 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.290058    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3188c970-5b86-4c03-95cc-12480aa11882-istio-data\") pod \"e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m\" (UID: \"3188c970-5b86-4c03-95cc-12480aa11882\") " pod="openshift-ingress/e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m"
Sep 14 09:57:21.290100 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.290102    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3188c970-5b86-4c03-95cc-12480aa11882-istio-envoy\") pod \"e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m\" (UID: \"3188c970-5b86-4c03-95cc-12480aa11882\") " pod="openshift-ingress/e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m"
Sep 14 09:57:21.290374 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.290132    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a68cc340-1e5b-4971-96cc-91a62af7d642-workload-certs\") pod \"e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn\" (UID: \"a68cc340-1e5b-4971-96cc-91a62af7d642\") " pod="openshift-ingress/e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn"
Sep 14 09:57:21.290374 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.290154    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3188c970-5b86-4c03-95cc-12480aa11882-istiod-ca-cert\") pod \"e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m\" (UID: \"3188c970-5b86-4c03-95cc-12480aa11882\") " pod="openshift-ingress/e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m"
Sep 14 09:57:21.290374 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.290179    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdp8m\" (UniqueName: \"kubernetes.io/projected/3188c970-5b86-4c03-95cc-12480aa11882-kube-api-access-mdp8m\") pod \"e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m\" (UID: \"3188c970-5b86-4c03-95cc-12480aa11882\") " pod="openshift-ingress/e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m"
Sep 14 09:57:21.290374 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.290196    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a68cc340-1e5b-4971-96cc-91a62af7d642-credential-socket\") pod \"e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn\" (UID: \"a68cc340-1e5b-4971-96cc-91a62af7d642\") " pod="openshift-ingress/e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn"
Sep 14 09:57:21.290374 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.290231    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4vbs\" (UniqueName: \"kubernetes.io/projected/a68cc340-1e5b-4971-96cc-91a62af7d642-kube-api-access-m4vbs\") pod \"e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn\" (UID: \"a68cc340-1e5b-4971-96cc-91a62af7d642\") " pod="openshift-ingress/e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn"
Sep 14 09:57:21.290637 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.290398    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3188c970-5b86-4c03-95cc-12480aa11882-istio-token\") pod \"e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m\" (UID: \"3188c970-5b86-4c03-95cc-12480aa11882\") " pod="openshift-ingress/e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m"
Sep 14 09:57:21.290637 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.290428    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3188c970-5b86-4c03-95cc-12480aa11882-istio-podinfo\") pod \"e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m\" (UID: \"3188c970-5b86-4c03-95cc-12480aa11882\") " pod="openshift-ingress/e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m"
Sep 14 09:57:21.290637 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.290565    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3188c970-5b86-4c03-95cc-12480aa11882-istio-data\") pod \"e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m\" (UID: \"3188c970-5b86-4c03-95cc-12480aa11882\") " pod="openshift-ingress/e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m"
Sep 14 09:57:21.290793 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.290657    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3188c970-5b86-4c03-95cc-12480aa11882-workload-socket\") pod \"e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m\" (UID: \"3188c970-5b86-4c03-95cc-12480aa11882\") " pod="openshift-ingress/e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m"
Sep 14 09:57:21.290793 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.290702    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3188c970-5b86-4c03-95cc-12480aa11882-workload-certs\") pod \"e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m\" (UID: \"3188c970-5b86-4c03-95cc-12480aa11882\") " pod="openshift-ingress/e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m"
Sep 14 09:57:21.290793 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.290779    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a68cc340-1e5b-4971-96cc-91a62af7d642-istio-envoy\") pod \"e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn\" (UID: \"a68cc340-1e5b-4971-96cc-91a62af7d642\") " pod="openshift-ingress/e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn"
Sep 14 09:57:21.290949 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.290810    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a68cc340-1e5b-4971-96cc-91a62af7d642-workload-socket\") pod \"e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn\" (UID: \"a68cc340-1e5b-4971-96cc-91a62af7d642\") " pod="openshift-ingress/e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn"
Sep 14 09:57:21.290949 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.290881    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a68cc340-1e5b-4971-96cc-91a62af7d642-istio-podinfo\") pod \"e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn\" (UID: \"a68cc340-1e5b-4971-96cc-91a62af7d642\") " pod="openshift-ingress/e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn"
Sep 14 09:57:21.290949 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.290902    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3188c970-5b86-4c03-95cc-12480aa11882-workload-socket\") pod \"e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m\" (UID: \"3188c970-5b86-4c03-95cc-12480aa11882\") " pod="openshift-ingress/e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m"
Sep 14 09:57:21.291136 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.290963    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3188c970-5b86-4c03-95cc-12480aa11882-istiod-ca-cert\") pod \"e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m\" (UID: \"3188c970-5b86-4c03-95cc-12480aa11882\") " pod="openshift-ingress/e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m"
Sep 14 09:57:21.291136 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.290987    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a68cc340-1e5b-4971-96cc-91a62af7d642-istio-data\") pod \"e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn\" (UID: \"a68cc340-1e5b-4971-96cc-91a62af7d642\") " pod="openshift-ingress/e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn"
Sep 14 09:57:21.291238 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.291142    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3188c970-5b86-4c03-95cc-12480aa11882-workload-certs\") pod \"e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m\" (UID: \"3188c970-5b86-4c03-95cc-12480aa11882\") " pod="openshift-ingress/e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m"
Sep 14 09:57:21.291293 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.291264    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a68cc340-1e5b-4971-96cc-91a62af7d642-istiod-ca-cert\") pod \"e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn\" (UID: \"a68cc340-1e5b-4971-96cc-91a62af7d642\") " pod="openshift-ingress/e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn"
Sep 14 09:57:21.291410 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.291313    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3188c970-5b86-4c03-95cc-12480aa11882-credential-socket\") pod \"e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m\" (UID: \"3188c970-5b86-4c03-95cc-12480aa11882\") " pod="openshift-ingress/e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m"
Sep 14 09:57:21.291410 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.291368    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a68cc340-1e5b-4971-96cc-91a62af7d642-istio-token\") pod \"e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn\" (UID: \"a68cc340-1e5b-4971-96cc-91a62af7d642\") " pod="openshift-ingress/e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn"
Sep 14 09:57:21.291675 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.291650    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3188c970-5b86-4c03-95cc-12480aa11882-credential-socket\") pod \"e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m\" (UID: \"3188c970-5b86-4c03-95cc-12480aa11882\") " pod="openshift-ingress/e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m"
Sep 14 09:57:21.292977 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.292958    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3188c970-5b86-4c03-95cc-12480aa11882-istio-envoy\") pod \"e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m\" (UID: \"3188c970-5b86-4c03-95cc-12480aa11882\") " pod="openshift-ingress/e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m"
Sep 14 09:57:21.293357 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.293332    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3188c970-5b86-4c03-95cc-12480aa11882-istio-podinfo\") pod \"e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m\" (UID: \"3188c970-5b86-4c03-95cc-12480aa11882\") " pod="openshift-ingress/e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m"
Sep 14 09:57:21.298676 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.298636    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw"]
Sep 14 09:57:21.306234 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.306197    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw"
Sep 14 09:57:21.306234 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.306227    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3188c970-5b86-4c03-95cc-12480aa11882-istio-token\") pod \"e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m\" (UID: \"3188c970-5b86-4c03-95cc-12480aa11882\") " pod="openshift-ingress/e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m"
Sep 14 09:57:21.306513 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.306490    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdp8m\" (UniqueName: \"kubernetes.io/projected/3188c970-5b86-4c03-95cc-12480aa11882-kube-api-access-mdp8m\") pod \"e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m\" (UID: \"3188c970-5b86-4c03-95cc-12480aa11882\") " pod="openshift-ingress/e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m"
Sep 14 09:57:21.308361 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.308333    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-b6812e-openshift-default-dockercfg-c7w77\""
Sep 14 09:57:21.315509 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.315471    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw"]
Sep 14 09:57:21.354221 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.354186    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:21.392875 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.392775    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9d150e6-194e-44be-8a5f-bb826032f664-credential-socket\") pod \"e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw\" (UID: \"a9d150e6-194e-44be-8a5f-bb826032f664\") " pod="openshift-ingress/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw"
Sep 14 09:57:21.392875 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.392832    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a68cc340-1e5b-4971-96cc-91a62af7d642-workload-certs\") pod \"e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn\" (UID: \"a68cc340-1e5b-4971-96cc-91a62af7d642\") " pod="openshift-ingress/e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn"
Sep 14 09:57:21.392875 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.392865    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a68cc340-1e5b-4971-96cc-91a62af7d642-credential-socket\") pod \"e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn\" (UID: \"a68cc340-1e5b-4971-96cc-91a62af7d642\") " pod="openshift-ingress/e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn"
Sep 14 09:57:21.393197 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.392891    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4vbs\" (UniqueName: \"kubernetes.io/projected/a68cc340-1e5b-4971-96cc-91a62af7d642-kube-api-access-m4vbs\") pod \"e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn\" (UID: \"a68cc340-1e5b-4971-96cc-91a62af7d642\") " pod="openshift-ingress/e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn"
Sep 14 09:57:21.393197 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.392923    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9d150e6-194e-44be-8a5f-bb826032f664-istiod-ca-cert\") pod \"e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw\" (UID: \"a9d150e6-194e-44be-8a5f-bb826032f664\") " pod="openshift-ingress/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw"
Sep 14 09:57:21.393349 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.393326    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a68cc340-1e5b-4971-96cc-91a62af7d642-workload-certs\") pod \"e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn\" (UID: \"a68cc340-1e5b-4971-96cc-91a62af7d642\") " pod="openshift-ingress/e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn"
Sep 14 09:57:21.393413 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.393329    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a68cc340-1e5b-4971-96cc-91a62af7d642-istio-envoy\") pod \"e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn\" (UID: \"a68cc340-1e5b-4971-96cc-91a62af7d642\") " pod="openshift-ingress/e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn"
Sep 14 09:57:21.393413 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.393375    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9d150e6-194e-44be-8a5f-bb826032f664-istio-token\") pod \"e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw\" (UID: \"a9d150e6-194e-44be-8a5f-bb826032f664\") " pod="openshift-ingress/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw"
Sep 14 09:57:21.393413 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.393377    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a68cc340-1e5b-4971-96cc-91a62af7d642-credential-socket\") pod \"e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn\" (UID: \"a68cc340-1e5b-4971-96cc-91a62af7d642\") " pod="openshift-ingress/e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn"
Sep 14 09:57:21.393413 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.393407    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a68cc340-1e5b-4971-96cc-91a62af7d642-workload-socket\") pod \"e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn\" (UID: \"a68cc340-1e5b-4971-96cc-91a62af7d642\") " pod="openshift-ingress/e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn"
Sep 14 09:57:21.393632 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.393436    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a68cc340-1e5b-4971-96cc-91a62af7d642-istio-podinfo\") pod \"e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn\" (UID: \"a68cc340-1e5b-4971-96cc-91a62af7d642\") " pod="openshift-ingress/e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn"
Sep 14 09:57:21.393632 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.393452    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9d150e6-194e-44be-8a5f-bb826032f664-workload-certs\") pod \"e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw\" (UID: \"a9d150e6-194e-44be-8a5f-bb826032f664\") " pod="openshift-ingress/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw"
Sep 14 09:57:21.393632 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.393473    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9d150e6-194e-44be-8a5f-bb826032f664-istio-envoy\") pod \"e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw\" (UID: \"a9d150e6-194e-44be-8a5f-bb826032f664\") " pod="openshift-ingress/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw"
Sep 14 09:57:21.393632 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.393514    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9d150e6-194e-44be-8a5f-bb826032f664-istio-podinfo\") pod \"e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw\" (UID: \"a9d150e6-194e-44be-8a5f-bb826032f664\") " pod="openshift-ingress/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw"
Sep 14 09:57:21.393632 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.393544    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a68cc340-1e5b-4971-96cc-91a62af7d642-istio-data\") pod \"e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn\" (UID: \"a68cc340-1e5b-4971-96cc-91a62af7d642\") " pod="openshift-ingress/e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn"
Sep 14 09:57:21.393632 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.393593    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a68cc340-1e5b-4971-96cc-91a62af7d642-istiod-ca-cert\") pod \"e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn\" (UID: \"a68cc340-1e5b-4971-96cc-91a62af7d642\") " pod="openshift-ingress/e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn"
Sep 14 09:57:21.393958 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.393637    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9d150e6-194e-44be-8a5f-bb826032f664-istio-data\") pod \"e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw\" (UID: \"a9d150e6-194e-44be-8a5f-bb826032f664\") " pod="openshift-ingress/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw"
Sep 14 09:57:21.393958 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.393676    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvkbs\" (UniqueName: \"kubernetes.io/projected/a9d150e6-194e-44be-8a5f-bb826032f664-kube-api-access-bvkbs\") pod \"e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw\" (UID: \"a9d150e6-194e-44be-8a5f-bb826032f664\") " pod="openshift-ingress/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw"
Sep 14 09:57:21.393958 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.393741    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a68cc340-1e5b-4971-96cc-91a62af7d642-istio-token\") pod \"e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn\" (UID: \"a68cc340-1e5b-4971-96cc-91a62af7d642\") " pod="openshift-ingress/e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn"
Sep 14 09:57:21.393958 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.393773    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9d150e6-194e-44be-8a5f-bb826032f664-workload-socket\") pod \"e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw\" (UID: \"a9d150e6-194e-44be-8a5f-bb826032f664\") " pod="openshift-ingress/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw"
Sep 14 09:57:21.393958 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.393867    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a68cc340-1e5b-4971-96cc-91a62af7d642-workload-socket\") pod \"e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn\" (UID: \"a68cc340-1e5b-4971-96cc-91a62af7d642\") " pod="openshift-ingress/e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn"
Sep 14 09:57:21.394255 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.394171    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a68cc340-1e5b-4971-96cc-91a62af7d642-istio-data\") pod \"e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn\" (UID: \"a68cc340-1e5b-4971-96cc-91a62af7d642\") " pod="openshift-ingress/e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn"
Sep 14 09:57:21.394481 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.394460    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a68cc340-1e5b-4971-96cc-91a62af7d642-istiod-ca-cert\") pod \"e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn\" (UID: \"a68cc340-1e5b-4971-96cc-91a62af7d642\") " pod="openshift-ingress/e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn"
Sep 14 09:57:21.395831 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.395809    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a68cc340-1e5b-4971-96cc-91a62af7d642-istio-envoy\") pod \"e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn\" (UID: \"a68cc340-1e5b-4971-96cc-91a62af7d642\") " pod="openshift-ingress/e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn"
Sep 14 09:57:21.396267 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.396244    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a68cc340-1e5b-4971-96cc-91a62af7d642-istio-podinfo\") pod \"e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn\" (UID: \"a68cc340-1e5b-4971-96cc-91a62af7d642\") " pod="openshift-ingress/e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn"
Sep 14 09:57:21.410137 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.409904    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4vbs\" (UniqueName: \"kubernetes.io/projected/a68cc340-1e5b-4971-96cc-91a62af7d642-kube-api-access-m4vbs\") pod \"e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn\" (UID: \"a68cc340-1e5b-4971-96cc-91a62af7d642\") " pod="openshift-ingress/e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn"
Sep 14 09:57:21.410137 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.410075    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a68cc340-1e5b-4971-96cc-91a62af7d642-istio-token\") pod \"e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn\" (UID: \"a68cc340-1e5b-4971-96cc-91a62af7d642\") " pod="openshift-ingress/e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn"
Sep 14 09:57:21.456186 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.456135    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m"
Sep 14 09:57:21.494294 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.494251    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9d150e6-194e-44be-8a5f-bb826032f664-istio-podinfo\") pod \"e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw\" (UID: \"a9d150e6-194e-44be-8a5f-bb826032f664\") " pod="openshift-ingress/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw"
Sep 14 09:57:21.494507 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.494319    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9d150e6-194e-44be-8a5f-bb826032f664-istio-data\") pod \"e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw\" (UID: \"a9d150e6-194e-44be-8a5f-bb826032f664\") " pod="openshift-ingress/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw"
Sep 14 09:57:21.494507 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.494345    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvkbs\" (UniqueName: \"kubernetes.io/projected/a9d150e6-194e-44be-8a5f-bb826032f664-kube-api-access-bvkbs\") pod \"e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw\" (UID: \"a9d150e6-194e-44be-8a5f-bb826032f664\") " pod="openshift-ingress/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw"
Sep 14 09:57:21.494507 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.494391    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9d150e6-194e-44be-8a5f-bb826032f664-workload-socket\") pod \"e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw\" (UID: \"a9d150e6-194e-44be-8a5f-bb826032f664\") " pod="openshift-ingress/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw"
Sep 14 09:57:21.494682 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.494622    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9d150e6-194e-44be-8a5f-bb826032f664-credential-socket\") pod \"e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw\" (UID: \"a9d150e6-194e-44be-8a5f-bb826032f664\") " pod="openshift-ingress/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw"
Sep 14 09:57:21.494810 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.494744    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9d150e6-194e-44be-8a5f-bb826032f664-istiod-ca-cert\") pod \"e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw\" (UID: \"a9d150e6-194e-44be-8a5f-bb826032f664\") " pod="openshift-ingress/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw"
Sep 14 09:57:21.494895 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.494834    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9d150e6-194e-44be-8a5f-bb826032f664-workload-socket\") pod \"e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw\" (UID: \"a9d150e6-194e-44be-8a5f-bb826032f664\") " pod="openshift-ingress/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw"
Sep 14 09:57:21.494895 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.494846    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9d150e6-194e-44be-8a5f-bb826032f664-istio-token\") pod \"e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw\" (UID: \"a9d150e6-194e-44be-8a5f-bb826032f664\") " pod="openshift-ingress/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw"
Sep 14 09:57:21.495045 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.494906    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9d150e6-194e-44be-8a5f-bb826032f664-workload-certs\") pod \"e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw\" (UID: \"a9d150e6-194e-44be-8a5f-bb826032f664\") " pod="openshift-ingress/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw"
Sep 14 09:57:21.495045 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.494924    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9d150e6-194e-44be-8a5f-bb826032f664-istio-data\") pod \"e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw\" (UID: \"a9d150e6-194e-44be-8a5f-bb826032f664\") " pod="openshift-ingress/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw"
Sep 14 09:57:21.495045 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.494931    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9d150e6-194e-44be-8a5f-bb826032f664-istio-envoy\") pod \"e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw\" (UID: \"a9d150e6-194e-44be-8a5f-bb826032f664\") " pod="openshift-ingress/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw"
Sep 14 09:57:21.495768 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.495134    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9d150e6-194e-44be-8a5f-bb826032f664-credential-socket\") pod \"e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw\" (UID: \"a9d150e6-194e-44be-8a5f-bb826032f664\") " pod="openshift-ingress/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw"
Sep 14 09:57:21.495768 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.495476    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9d150e6-194e-44be-8a5f-bb826032f664-workload-certs\") pod \"e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw\" (UID: \"a9d150e6-194e-44be-8a5f-bb826032f664\") " pod="openshift-ingress/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw"
Sep 14 09:57:21.495768 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.495521    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9d150e6-194e-44be-8a5f-bb826032f664-istiod-ca-cert\") pod \"e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw\" (UID: \"a9d150e6-194e-44be-8a5f-bb826032f664\") " pod="openshift-ingress/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw"
Sep 14 09:57:21.497832 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.497773    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9d150e6-194e-44be-8a5f-bb826032f664-istio-envoy\") pod \"e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw\" (UID: \"a9d150e6-194e-44be-8a5f-bb826032f664\") " pod="openshift-ingress/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw"
Sep 14 09:57:21.498535 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.498505    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9d150e6-194e-44be-8a5f-bb826032f664-istio-podinfo\") pod \"e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw\" (UID: \"a9d150e6-194e-44be-8a5f-bb826032f664\") " pod="openshift-ingress/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw"
Sep 14 09:57:21.505137 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.505105    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9d150e6-194e-44be-8a5f-bb826032f664-istio-token\") pod \"e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw\" (UID: \"a9d150e6-194e-44be-8a5f-bb826032f664\") " pod="openshift-ingress/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw"
Sep 14 09:57:21.505305 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.505203    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvkbs\" (UniqueName: \"kubernetes.io/projected/a9d150e6-194e-44be-8a5f-bb826032f664-kube-api-access-bvkbs\") pod \"e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw\" (UID: \"a9d150e6-194e-44be-8a5f-bb826032f664\") " pod="openshift-ingress/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw"
Sep 14 09:57:21.529231 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.529192    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn"
Sep 14 09:57:21.546942 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.546905    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:21.628351 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.628312    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw"
Sep 14 09:57:21.647622 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.647585    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:21.746186 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.746147    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m"]
Sep 14 09:57:21.748789 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:57:21.748762    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3188c970_5b86_4c03_95cc_12480aa11882.slice/crio-757dcf3abd95c2a95f1635b44d20ab000e7d8a1e8cf5294f69c0a0e7c1dcc234 WatchSource:0}: Error finding container 757dcf3abd95c2a95f1635b44d20ab000e7d8a1e8cf5294f69c0a0e7c1dcc234: Status 404 returned error can't find the container with id 757dcf3abd95c2a95f1635b44d20ab000e7d8a1e8cf5294f69c0a0e7c1dcc234
Sep 14 09:57:21.750974 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.750939    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 09:57:21.751083 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.751063    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 09:57:21.751121 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.751109    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 09:57:21.821696 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.821662    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn"]
Sep 14 09:57:21.822576 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:57:21.822548    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda68cc340_1e5b_4971_96cc_91a62af7d642.slice/crio-6cfa0b66e2deb2384a8f7b249708dec1f40ecfd6f7a8e80dea0f227eba907eb3 WatchSource:0}: Error finding container 6cfa0b66e2deb2384a8f7b249708dec1f40ecfd6f7a8e80dea0f227eba907eb3: Status 404 returned error can't find the container with id 6cfa0b66e2deb2384a8f7b249708dec1f40ecfd6f7a8e80dea0f227eba907eb3
Sep 14 09:57:21.825271 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.825224    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 09:57:21.825374 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.825329    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 09:57:21.825431 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.825376    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 09:57:21.916042 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.915934    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw"]
Sep 14 09:57:21.920474 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:57:21.920429    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9d150e6_194e_44be_8a5f_bb826032f664.slice/crio-a8bce98008164e3c02bba4ecad6e48ab5d732a8ef7135f97c4b628876c276f12 WatchSource:0}: Error finding container a8bce98008164e3c02bba4ecad6e48ab5d732a8ef7135f97c4b628876c276f12: Status 404 returned error can't find the container with id a8bce98008164e3c02bba4ecad6e48ab5d732a8ef7135f97c4b628876c276f12
Sep 14 09:57:21.923087 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.923039    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 09:57:21.923194 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.923151    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 09:57:21.923248 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:21.923200    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 09:57:22.242037 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:22.241954    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:22.357990 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:22.357939    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:22.665348 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:22.665256    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:22.755133 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:22.755100    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7690-25321d"]
Sep 14 09:57:22.758980 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:22.758947    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn" event={"ID":"a68cc340-1e5b-4971-96cc-91a62af7d642","Type":"ContainerStarted","Data":"2582027d20843d4e4febcf69ccbb6c0043ba638676372fb2b834cfa4cea62f31"}
Sep 14 09:57:22.759197 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:22.758990    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn" event={"ID":"a68cc340-1e5b-4971-96cc-91a62af7d642","Type":"ContainerStarted","Data":"6cfa0b66e2deb2384a8f7b249708dec1f40ecfd6f7a8e80dea0f227eba907eb3"}
Sep 14 09:57:22.759197 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:22.759030    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw" event={"ID":"a9d150e6-194e-44be-8a5f-bb826032f664","Type":"ContainerStarted","Data":"d42c378f47c1e6616509d23a6f306ae3a147e0c5e1bae984a4774ac1fe3aa63e"}
Sep 14 09:57:22.759197 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:22.759053    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw" event={"ID":"a9d150e6-194e-44be-8a5f-bb826032f664","Type":"ContainerStarted","Data":"a8bce98008164e3c02bba4ecad6e48ab5d732a8ef7135f97c4b628876c276f12"}
Sep 14 09:57:22.759197 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:22.759074    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7690-25321d"
Sep 14 09:57:22.759197 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:22.759191    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m" event={"ID":"3188c970-5b86-4c03-95cc-12480aa11882","Type":"ContainerStarted","Data":"f7e0b1923c1515b6523444260d8e8f86a71e1c4c8e8a06270c6cbc3fa374ab22"}
Sep 14 09:57:22.759394 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:22.759224    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m" event={"ID":"3188c970-5b86-4c03-95cc-12480aa11882","Type":"ContainerStarted","Data":"757dcf3abd95c2a95f1635b44d20ab000e7d8a1e8cf5294f69c0a0e7c1dcc234"}
Sep 14 09:57:22.762133 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:22.762102    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-hjc9w\""
Sep 14 09:57:22.765540 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:22.765421    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7690-25321d"]
Sep 14 09:57:22.783894 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:22.783835    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn" podStartSLOduration=1.7838154990000001 podStartE2EDuration="1.783815499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:57:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:57:22.782315634 +0000 UTC m=+831.821990258" watchObservedRunningTime="2026-09-14 09:57:22.783815499 +0000 UTC m=+831.823490111"
Sep 14 09:57:22.805499 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:22.805440    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m" podStartSLOduration=1.805419987 podStartE2EDuration="1.805419987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:57:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:57:22.803503681 +0000 UTC m=+831.843178299" watchObservedRunningTime="2026-09-14 09:57:22.805419987 +0000 UTC m=+831.845094608"
Sep 14 09:57:22.808347 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:22.808296    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jjzz\" (UniqueName: \"kubernetes.io/projected/777a7bd8-27f6-4981-953f-0a8f98caab38-kube-api-access-8jjzz\") pod \"test-curl-7690-25321d\" (UID: \"777a7bd8-27f6-4981-953f-0a8f98caab38\") " pod="opendatahub/test-curl-7690-25321d"
Sep 14 09:57:22.828928 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:22.828617    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw" podStartSLOduration=1.828591522 podStartE2EDuration="1.828591522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:57:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:57:22.827695479 +0000 UTC m=+831.867370101" watchObservedRunningTime="2026-09-14 09:57:22.828591522 +0000 UTC m=+831.868266166"
Sep 14 09:57:22.909622 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:22.909590    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jjzz\" (UniqueName: \"kubernetes.io/projected/777a7bd8-27f6-4981-953f-0a8f98caab38-kube-api-access-8jjzz\") pod \"test-curl-7690-25321d\" (UID: \"777a7bd8-27f6-4981-953f-0a8f98caab38\") " pod="opendatahub/test-curl-7690-25321d"
Sep 14 09:57:22.918237 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:22.918156    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jjzz\" (UniqueName: \"kubernetes.io/projected/777a7bd8-27f6-4981-953f-0a8f98caab38-kube-api-access-8jjzz\") pod \"test-curl-7690-25321d\" (UID: \"777a7bd8-27f6-4981-953f-0a8f98caab38\") " pod="opendatahub/test-curl-7690-25321d"
Sep 14 09:57:23.072523 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:23.072479    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7690-25321d"
Sep 14 09:57:23.356145 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:23.356115    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7690-25321d"]
Sep 14 09:57:23.358593 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:57:23.358557    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod777a7bd8_27f6_4981_953f_0a8f98caab38.slice/crio-8fe4178c6a80adc0e40cdbda1b890a5d579560d030cc6333c4f19360402ec80d WatchSource:0}: Error finding container 8fe4178c6a80adc0e40cdbda1b890a5d579560d030cc6333c4f19360402ec80d: Status 404 returned error can't find the container with id 8fe4178c6a80adc0e40cdbda1b890a5d579560d030cc6333c4f19360402ec80d
Sep 14 09:57:23.456659 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:23.456615    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m"
Sep 14 09:57:23.461971 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:23.461939    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m"
Sep 14 09:57:23.530218 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:23.530171    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn"
Sep 14 09:57:23.535549 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:23.535516    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn"
Sep 14 09:57:23.629495 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:23.629402    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw"
Sep 14 09:57:23.635134 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:23.635100    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw"
Sep 14 09:57:23.766703 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:23.766660    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7690-25321d" event={"ID":"777a7bd8-27f6-4981-953f-0a8f98caab38","Type":"ContainerStarted","Data":"8fe4178c6a80adc0e40cdbda1b890a5d579560d030cc6333c4f19360402ec80d"}
Sep 14 09:57:23.767018 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:23.766979    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw"
Sep 14 09:57:23.767527 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:23.767504    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn"
Sep 14 09:57:23.767659 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:23.767547    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m"
Sep 14 09:57:23.768186 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:23.768151    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn"
Sep 14 09:57:23.768338 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:23.768277    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m"
Sep 14 09:57:23.768338 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:23.768336    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw"
Sep 14 09:57:24.873130 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:24.873090    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:24.985475 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:24.985439    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:25.097395 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:25.097359    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:25.190318 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:25.190231    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:25.348099 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:25.348048    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:25.467415 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:25.467328    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:25.781474 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:25.781431    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7690-25321d" event={"ID":"777a7bd8-27f6-4981-953f-0a8f98caab38","Type":"ContainerStarted","Data":"673d4aa18d5dd0bdaae40abaae7a2d8c1424bdb2d37a4c5c7727114ffa3fff30"}
Sep 14 09:57:25.807155 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:25.807082    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7690-25321d" podStartSLOduration=1.5488379380000001 podStartE2EDuration="3.807064471s" totalImagesPullingTime="2.258226533s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:57:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:57:25.804841598 +0000 UTC m=+834.844516219" watchObservedRunningTime="2026-09-14 09:57:25.807064471 +0000 UTC m=+834.846739091"
Sep 14 09:57:26.126336 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:26.126234    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:26.230801 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:26.230766    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:26.335220 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:26.335185    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:26.424207 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:26.424113    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7690-25321d"]
Sep 14 09:57:26.434175 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:26.434138    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7690-25321d"]
Sep 14 09:57:26.554252 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:26.554216    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7690-f9bf0c"]
Sep 14 09:57:26.554633 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:26.554619    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="777a7bd8-27f6-4981-953f-0a8f98caab38" containerName="test-curl-7690-25321d"
Sep 14 09:57:26.554684 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:26.554635    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="777a7bd8-27f6-4981-953f-0a8f98caab38" containerName="test-curl-7690-25321d"
Sep 14 09:57:26.554719 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:26.554706    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="777a7bd8-27f6-4981-953f-0a8f98caab38" containerName="test-curl-7690-25321d"
Sep 14 09:57:26.558389 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:26.558363    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7690-f9bf0c"
Sep 14 09:57:26.572928 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:26.572891    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7690-f9bf0c"]
Sep 14 09:57:26.646618 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:26.646581    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s964\" (UniqueName: \"kubernetes.io/projected/1dcbc0d5-6f08-4f01-a7c5-7300773fc6c3-kube-api-access-5s964\") pod \"test-curl-7690-f9bf0c\" (UID: \"1dcbc0d5-6f08-4f01-a7c5-7300773fc6c3\") " pod="opendatahub/test-curl-7690-f9bf0c"
Sep 14 09:57:26.747665 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:26.747623    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s964\" (UniqueName: \"kubernetes.io/projected/1dcbc0d5-6f08-4f01-a7c5-7300773fc6c3-kube-api-access-5s964\") pod \"test-curl-7690-f9bf0c\" (UID: \"1dcbc0d5-6f08-4f01-a7c5-7300773fc6c3\") " pod="opendatahub/test-curl-7690-f9bf0c"
Sep 14 09:57:26.758567 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:26.758536    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s964\" (UniqueName: \"kubernetes.io/projected/1dcbc0d5-6f08-4f01-a7c5-7300773fc6c3-kube-api-access-5s964\") pod \"test-curl-7690-f9bf0c\" (UID: \"1dcbc0d5-6f08-4f01-a7c5-7300773fc6c3\") " pod="opendatahub/test-curl-7690-f9bf0c"
Sep 14 09:57:26.787269 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:57:26.787233    2563 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7690-25321d" container="test-curl-7690-25321d"
Sep 14 09:57:26.868620 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:26.868572    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7690-f9bf0c"
Sep 14 09:57:26.975038 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:26.974986    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:27.068252 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:27.068168    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:27.156678 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:27.156515    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7690-f9bf0c"]
Sep 14 09:57:27.160125 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:57:27.160090    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1dcbc0d5_6f08_4f01_a7c5_7300773fc6c3.slice/crio-c51c651d45029ce2f7d96f170cbc30a97a909404c8be90a756373fe4d0b2daf5 WatchSource:0}: Error finding container c51c651d45029ce2f7d96f170cbc30a97a909404c8be90a756373fe4d0b2daf5: Status 404 returned error can't find the container with id c51c651d45029ce2f7d96f170cbc30a97a909404c8be90a756373fe4d0b2daf5
Sep 14 09:57:27.507603 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:27.507562    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:27.607548 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:27.607513    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:27.794856 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:27.794750    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7690-f9bf0c" event={"ID":"1dcbc0d5-6f08-4f01-a7c5-7300773fc6c3","Type":"ContainerStarted","Data":"bf10e2400bd1877216fb67f0938a61056ab5badd11782237bb2c95552bb0fcf3"}
Sep 14 09:57:27.794856 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:27.794806    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7690-f9bf0c" event={"ID":"1dcbc0d5-6f08-4f01-a7c5-7300773fc6c3","Type":"ContainerStarted","Data":"c51c651d45029ce2f7d96f170cbc30a97a909404c8be90a756373fe4d0b2daf5"}
Sep 14 09:57:27.795185 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:27.794917    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7690-25321d" podUID="777a7bd8-27f6-4981-953f-0a8f98caab38" containerName="test-curl-7690-25321d" containerID="cri-o://673d4aa18d5dd0bdaae40abaae7a2d8c1424bdb2d37a4c5c7727114ffa3fff30" gracePeriod=2
Sep 14 09:57:27.810779 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:27.810700    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7690-f9bf0c" podStartSLOduration=1.81067781 podStartE2EDuration="1.81067781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:57:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:57:27.810525896 +0000 UTC m=+836.850200514" watchObservedRunningTime="2026-09-14 09:57:27.81067781 +0000 UTC m=+836.850352433"
Sep 14 09:57:28.203863 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:28.203781    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:28.304716 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:28.304663    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:28.472959 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:28.472867    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7690-f9bf0c"]
Sep 14 09:57:28.476896 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:28.476829    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7690-f9bf0c"]
Sep 14 09:57:28.800911 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:57:28.800876    2563 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7690-f9bf0c" container="test-curl-7690-f9bf0c"
Sep 14 09:57:28.874011 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:28.873970    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:28.979037 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:28.978975    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:29.075376 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:29.075269    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:29.229847 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:29.229814    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:29.387444 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:29.387322    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn"]
Sep 14 09:57:29.387735 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:29.387720    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dcbc0d5-6f08-4f01-a7c5-7300773fc6c3" containerName="test-curl-7690-f9bf0c"
Sep 14 09:57:29.387735 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:29.387736    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dcbc0d5-6f08-4f01-a7c5-7300773fc6c3" containerName="test-curl-7690-f9bf0c"
Sep 14 09:57:29.387856 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:29.387846    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dcbc0d5-6f08-4f01-a7c5-7300773fc6c3" containerName="test-curl-7690-f9bf0c"
Sep 14 09:57:29.391069 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:29.391044    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn"
Sep 14 09:57:29.395140 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:29.395110    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-8c98aa-openshift-default-dockercfg-rdqcn\""
Sep 14 09:57:29.416958 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:29.416920    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn"]
Sep 14 09:57:29.447409 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:29.447374    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:29.479764 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:29.479715    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eae8537b-56e1-4d2a-a43b-de5bb82f78b8-workload-socket\") pod \"e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn\" (UID: \"eae8537b-56e1-4d2a-a43b-de5bb82f78b8\") " pod="openshift-ingress/e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn"
Sep 14 09:57:29.479981 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:29.479795    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eae8537b-56e1-4d2a-a43b-de5bb82f78b8-istio-data\") pod \"e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn\" (UID: \"eae8537b-56e1-4d2a-a43b-de5bb82f78b8\") " pod="openshift-ingress/e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn"
Sep 14 09:57:29.479981 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:29.479835    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eae8537b-56e1-4d2a-a43b-de5bb82f78b8-istio-token\") pod \"e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn\" (UID: \"eae8537b-56e1-4d2a-a43b-de5bb82f78b8\") " pod="openshift-ingress/e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn"
Sep 14 09:57:29.479981 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:29.479874    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eae8537b-56e1-4d2a-a43b-de5bb82f78b8-istio-podinfo\") pod \"e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn\" (UID: \"eae8537b-56e1-4d2a-a43b-de5bb82f78b8\") " pod="openshift-ingress/e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn"
Sep 14 09:57:29.479981 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:29.479962    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eae8537b-56e1-4d2a-a43b-de5bb82f78b8-workload-certs\") pod \"e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn\" (UID: \"eae8537b-56e1-4d2a-a43b-de5bb82f78b8\") " pod="openshift-ingress/e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn"
Sep 14 09:57:29.480254 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:29.480068    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eae8537b-56e1-4d2a-a43b-de5bb82f78b8-istiod-ca-cert\") pod \"e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn\" (UID: \"eae8537b-56e1-4d2a-a43b-de5bb82f78b8\") " pod="openshift-ingress/e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn"
Sep 14 09:57:29.480254 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:29.480122    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eae8537b-56e1-4d2a-a43b-de5bb82f78b8-istio-envoy\") pod \"e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn\" (UID: \"eae8537b-56e1-4d2a-a43b-de5bb82f78b8\") " pod="openshift-ingress/e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn"
Sep 14 09:57:29.480254 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:29.480154    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g565h\" (UniqueName: \"kubernetes.io/projected/eae8537b-56e1-4d2a-a43b-de5bb82f78b8-kube-api-access-g565h\") pod \"e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn\" (UID: \"eae8537b-56e1-4d2a-a43b-de5bb82f78b8\") " pod="openshift-ingress/e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn"
Sep 14 09:57:29.480254 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:29.480203    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eae8537b-56e1-4d2a-a43b-de5bb82f78b8-credential-socket\") pod \"e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn\" (UID: \"eae8537b-56e1-4d2a-a43b-de5bb82f78b8\") " pod="openshift-ingress/e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn"
Sep 14 09:57:29.581548 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:29.581494    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eae8537b-56e1-4d2a-a43b-de5bb82f78b8-istiod-ca-cert\") pod \"e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn\" (UID: \"eae8537b-56e1-4d2a-a43b-de5bb82f78b8\") " pod="openshift-ingress/e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn"
Sep 14 09:57:29.581548 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:29.581555    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eae8537b-56e1-4d2a-a43b-de5bb82f78b8-istio-envoy\") pod \"e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn\" (UID: \"eae8537b-56e1-4d2a-a43b-de5bb82f78b8\") " pod="openshift-ingress/e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn"
Sep 14 09:57:29.581831 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:29.581581    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g565h\" (UniqueName: \"kubernetes.io/projected/eae8537b-56e1-4d2a-a43b-de5bb82f78b8-kube-api-access-g565h\") pod \"e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn\" (UID: \"eae8537b-56e1-4d2a-a43b-de5bb82f78b8\") " pod="openshift-ingress/e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn"
Sep 14 09:57:29.581831 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:29.581625    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eae8537b-56e1-4d2a-a43b-de5bb82f78b8-credential-socket\") pod \"e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn\" (UID: \"eae8537b-56e1-4d2a-a43b-de5bb82f78b8\") " pod="openshift-ingress/e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn"
Sep 14 09:57:29.581831 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:29.581678    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eae8537b-56e1-4d2a-a43b-de5bb82f78b8-workload-socket\") pod \"e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn\" (UID: \"eae8537b-56e1-4d2a-a43b-de5bb82f78b8\") " pod="openshift-ingress/e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn"
Sep 14 09:57:29.581831 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:29.581704    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eae8537b-56e1-4d2a-a43b-de5bb82f78b8-istio-data\") pod \"e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn\" (UID: \"eae8537b-56e1-4d2a-a43b-de5bb82f78b8\") " pod="openshift-ingress/e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn"
Sep 14 09:57:29.582080 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:29.582030    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eae8537b-56e1-4d2a-a43b-de5bb82f78b8-istio-token\") pod \"e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn\" (UID: \"eae8537b-56e1-4d2a-a43b-de5bb82f78b8\") " pod="openshift-ingress/e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn"
Sep 14 09:57:29.582137 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:29.582088    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eae8537b-56e1-4d2a-a43b-de5bb82f78b8-istio-podinfo\") pod \"e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn\" (UID: \"eae8537b-56e1-4d2a-a43b-de5bb82f78b8\") " pod="openshift-ingress/e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn"
Sep 14 09:57:29.582137 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:29.582107    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eae8537b-56e1-4d2a-a43b-de5bb82f78b8-istio-data\") pod \"e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn\" (UID: \"eae8537b-56e1-4d2a-a43b-de5bb82f78b8\") " pod="openshift-ingress/e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn"
Sep 14 09:57:29.582249 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:29.582147    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eae8537b-56e1-4d2a-a43b-de5bb82f78b8-credential-socket\") pod \"e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn\" (UID: \"eae8537b-56e1-4d2a-a43b-de5bb82f78b8\") " pod="openshift-ingress/e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn"
Sep 14 09:57:29.582249 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:29.582194    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eae8537b-56e1-4d2a-a43b-de5bb82f78b8-workload-certs\") pod \"e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn\" (UID: \"eae8537b-56e1-4d2a-a43b-de5bb82f78b8\") " pod="openshift-ingress/e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn"
Sep 14 09:57:29.582249 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:29.582213    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eae8537b-56e1-4d2a-a43b-de5bb82f78b8-workload-socket\") pod \"e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn\" (UID: \"eae8537b-56e1-4d2a-a43b-de5bb82f78b8\") " pod="openshift-ingress/e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn"
Sep 14 09:57:29.582412 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:29.582376    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eae8537b-56e1-4d2a-a43b-de5bb82f78b8-istiod-ca-cert\") pod \"e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn\" (UID: \"eae8537b-56e1-4d2a-a43b-de5bb82f78b8\") " pod="openshift-ingress/e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn"
Sep 14 09:57:29.582412 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:29.582407    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eae8537b-56e1-4d2a-a43b-de5bb82f78b8-workload-certs\") pod \"e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn\" (UID: \"eae8537b-56e1-4d2a-a43b-de5bb82f78b8\") " pod="openshift-ingress/e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn"
Sep 14 09:57:29.584158 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:29.584122    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eae8537b-56e1-4d2a-a43b-de5bb82f78b8-istio-envoy\") pod \"e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn\" (UID: \"eae8537b-56e1-4d2a-a43b-de5bb82f78b8\") " pod="openshift-ingress/e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn"
Sep 14 09:57:29.584389 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:29.584372    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eae8537b-56e1-4d2a-a43b-de5bb82f78b8-istio-podinfo\") pod \"e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn\" (UID: \"eae8537b-56e1-4d2a-a43b-de5bb82f78b8\") " pod="openshift-ingress/e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn"
Sep 14 09:57:29.593071 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:29.592981    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eae8537b-56e1-4d2a-a43b-de5bb82f78b8-istio-token\") pod \"e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn\" (UID: \"eae8537b-56e1-4d2a-a43b-de5bb82f78b8\") " pod="openshift-ingress/e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn"
Sep 14 09:57:29.599296 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:29.599264    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g565h\" (UniqueName: \"kubernetes.io/projected/eae8537b-56e1-4d2a-a43b-de5bb82f78b8-kube-api-access-g565h\") pod \"e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn\" (UID: \"eae8537b-56e1-4d2a-a43b-de5bb82f78b8\") " pod="openshift-ingress/e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn"
Sep 14 09:57:29.602128 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:29.602096    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:29.697403 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:29.697306    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:29.702334 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:29.702288    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn"
Sep 14 09:57:29.804481 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:29.804452    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:29.806948 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:29.806903    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7690-f9bf0c" podUID="1dcbc0d5-6f08-4f01-a7c5-7300773fc6c3" containerName="test-curl-7690-f9bf0c" containerID="cri-o://bf10e2400bd1877216fb67f0938a61056ab5badd11782237bb2c95552bb0fcf3" gracePeriod=2
Sep 14 09:57:29.986291 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:29.986257    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7690-25321d"
Sep 14 09:57:30.031316 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:30.031285    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn"]
Sep 14 09:57:30.033593 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:57:30.033560    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeae8537b_56e1_4d2a_a43b_de5bb82f78b8.slice/crio-4c123d948cfd934575fca0e26e66bf66960deb4fa81e8f7aab09506a96da4df4 WatchSource:0}: Error finding container 4c123d948cfd934575fca0e26e66bf66960deb4fa81e8f7aab09506a96da4df4: Status 404 returned error can't find the container with id 4c123d948cfd934575fca0e26e66bf66960deb4fa81e8f7aab09506a96da4df4
Sep 14 09:57:30.036037 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:30.035991    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 09:57:30.036112 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:30.036087    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 09:57:30.036161 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:30.036118    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 09:57:30.087436 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:30.087399    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8jjzz\" (UniqueName: \"kubernetes.io/projected/777a7bd8-27f6-4981-953f-0a8f98caab38-kube-api-access-8jjzz\") pod \"777a7bd8-27f6-4981-953f-0a8f98caab38\" (UID: \"777a7bd8-27f6-4981-953f-0a8f98caab38\") "
Sep 14 09:57:30.091027 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:30.090941    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/777a7bd8-27f6-4981-953f-0a8f98caab38-kube-api-access-8jjzz" (OuterVolumeSpecName: "kube-api-access-8jjzz") pod "777a7bd8-27f6-4981-953f-0a8f98caab38" (UID: "777a7bd8-27f6-4981-953f-0a8f98caab38"). InnerVolumeSpecName "kube-api-access-8jjzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 09:57:30.188446 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:30.188398    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jjzz\" (UniqueName: \"kubernetes.io/projected/777a7bd8-27f6-4981-953f-0a8f98caab38-kube-api-access-8jjzz\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:57:30.349130 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:30.349098    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:30.814964 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:30.814929    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn" event={"ID":"eae8537b-56e1-4d2a-a43b-de5bb82f78b8","Type":"ContainerStarted","Data":"8afcbeade6f146478a38713532fae180f6a9baa54dfaa533506ce9ad77f83340"}
Sep 14 09:57:30.815188 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:30.814993    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn" event={"ID":"eae8537b-56e1-4d2a-a43b-de5bb82f78b8","Type":"ContainerStarted","Data":"4c123d948cfd934575fca0e26e66bf66960deb4fa81e8f7aab09506a96da4df4"}
Sep 14 09:57:30.816512 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:30.816479    2563 generic.go:356] "Generic (PLEG): container finished" podID="777a7bd8-27f6-4981-953f-0a8f98caab38" containerID="673d4aa18d5dd0bdaae40abaae7a2d8c1424bdb2d37a4c5c7727114ffa3fff30" exitCode=137
Sep 14 09:57:30.816675 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:30.816520    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7690-25321d"
Sep 14 09:57:30.816675 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:30.816550    2563 scope.go:117] "RemoveContainer" containerID="673d4aa18d5dd0bdaae40abaae7a2d8c1424bdb2d37a4c5c7727114ffa3fff30"
Sep 14 09:57:30.827015 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:30.826969    2563 scope.go:117] "RemoveContainer" containerID="673d4aa18d5dd0bdaae40abaae7a2d8c1424bdb2d37a4c5c7727114ffa3fff30"
Sep 14 09:57:30.827409 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:57:30.827378    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"673d4aa18d5dd0bdaae40abaae7a2d8c1424bdb2d37a4c5c7727114ffa3fff30\": container with ID starting with 673d4aa18d5dd0bdaae40abaae7a2d8c1424bdb2d37a4c5c7727114ffa3fff30 not found: ID does not exist" containerID="673d4aa18d5dd0bdaae40abaae7a2d8c1424bdb2d37a4c5c7727114ffa3fff30"
Sep 14 09:57:30.827510 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:30.827421    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"673d4aa18d5dd0bdaae40abaae7a2d8c1424bdb2d37a4c5c7727114ffa3fff30"} err="failed to get container status \"673d4aa18d5dd0bdaae40abaae7a2d8c1424bdb2d37a4c5c7727114ffa3fff30\": rpc error: code = NotFound desc = could not find container \"673d4aa18d5dd0bdaae40abaae7a2d8c1424bdb2d37a4c5c7727114ffa3fff30\": container with ID starting with 673d4aa18d5dd0bdaae40abaae7a2d8c1424bdb2d37a4c5c7727114ffa3fff30 not found: ID does not exist"
Sep 14 09:57:30.842942 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:57:30.842889    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7690-25321d\" is forbidden: User \"system:node:ip-10-0-132-4.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-4.ec2.internal' and this object" podUID="777a7bd8-27f6-4981-953f-0a8f98caab38" pod="opendatahub/test-curl-7690-25321d"
Sep 14 09:57:30.844068 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:30.844018    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn" podStartSLOduration=1.843985585 podStartE2EDuration="1.843985585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:57:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:57:30.841126087 +0000 UTC m=+839.880800733" watchObservedRunningTime="2026-09-14 09:57:30.843985585 +0000 UTC m=+839.883660219"
Sep 14 09:57:31.481979 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:57:31.481942    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7690-25321d\" is forbidden: User \"system:node:ip-10-0-132-4.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-4.ec2.internal' and this object" podUID="777a7bd8-27f6-4981-953f-0a8f98caab38" pod="opendatahub/test-curl-7690-25321d"
Sep 14 09:57:31.483975 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:31.483945    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="777a7bd8-27f6-4981-953f-0a8f98caab38" path="/var/lib/kubelet/pods/777a7bd8-27f6-4981-953f-0a8f98caab38/volumes"
Sep 14 09:57:31.494402 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:31.494364    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:31.702986 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:31.702888    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn"
Sep 14 09:57:31.708874 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:31.708841    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn"
Sep 14 09:57:31.822245 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:31.822219    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn"
Sep 14 09:57:31.823412 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:31.823387    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn"
Sep 14 09:57:32.004919 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:32.004886    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7690-f9bf0c"
Sep 14 09:57:32.110018 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:32.109970    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5s964\" (UniqueName: \"kubernetes.io/projected/1dcbc0d5-6f08-4f01-a7c5-7300773fc6c3-kube-api-access-5s964\") pod \"1dcbc0d5-6f08-4f01-a7c5-7300773fc6c3\" (UID: \"1dcbc0d5-6f08-4f01-a7c5-7300773fc6c3\") "
Sep 14 09:57:32.112186 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:32.112154    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dcbc0d5-6f08-4f01-a7c5-7300773fc6c3-kube-api-access-5s964" (OuterVolumeSpecName: "kube-api-access-5s964") pod "1dcbc0d5-6f08-4f01-a7c5-7300773fc6c3" (UID: "1dcbc0d5-6f08-4f01-a7c5-7300773fc6c3"). InnerVolumeSpecName "kube-api-access-5s964". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 09:57:32.210807 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:32.210766    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5s964\" (UniqueName: \"kubernetes.io/projected/1dcbc0d5-6f08-4f01-a7c5-7300773fc6c3-kube-api-access-5s964\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:57:32.349931 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:32.349879    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:32.449201 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:32.449150    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:32.549958 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:32.549929    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:32.649963 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:32.649875    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:32.828704 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:32.828665    2563 generic.go:356] "Generic (PLEG): container finished" podID="1dcbc0d5-6f08-4f01-a7c5-7300773fc6c3" containerID="bf10e2400bd1877216fb67f0938a61056ab5badd11782237bb2c95552bb0fcf3" exitCode=137
Sep 14 09:57:32.828704 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:32.828709    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7690-f9bf0c"
Sep 14 09:57:32.828937 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:32.828760    2563 scope.go:117] "RemoveContainer" containerID="bf10e2400bd1877216fb67f0938a61056ab5badd11782237bb2c95552bb0fcf3"
Sep 14 09:57:32.844939 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:57:32.844896    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7690-f9bf0c\" is forbidden: User \"system:node:ip-10-0-132-4.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-4.ec2.internal' and this object" podUID="1dcbc0d5-6f08-4f01-a7c5-7300773fc6c3" pod="opendatahub/test-curl-7690-f9bf0c"
Sep 14 09:57:32.846326 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:32.846294    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:32.848751 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:32.848648    2563 scope.go:117] "RemoveContainer" containerID="bf10e2400bd1877216fb67f0938a61056ab5badd11782237bb2c95552bb0fcf3"
Sep 14 09:57:32.849079 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:57:32.849058    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf10e2400bd1877216fb67f0938a61056ab5badd11782237bb2c95552bb0fcf3\": container with ID starting with bf10e2400bd1877216fb67f0938a61056ab5badd11782237bb2c95552bb0fcf3 not found: ID does not exist" containerID="bf10e2400bd1877216fb67f0938a61056ab5badd11782237bb2c95552bb0fcf3"
Sep 14 09:57:32.849150 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:32.849087    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf10e2400bd1877216fb67f0938a61056ab5badd11782237bb2c95552bb0fcf3"} err="failed to get container status \"bf10e2400bd1877216fb67f0938a61056ab5badd11782237bb2c95552bb0fcf3\": rpc error: code = NotFound desc = could not find container \"bf10e2400bd1877216fb67f0938a61056ab5badd11782237bb2c95552bb0fcf3\": container with ID starting with bf10e2400bd1877216fb67f0938a61056ab5badd11782237bb2c95552bb0fcf3 not found: ID does not exist"
Sep 14 09:57:32.854871 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:32.854837    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk"]
Sep 14 09:57:32.859758 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:32.859732    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk"
Sep 14 09:57:32.862303 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:32.862279    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-9eed367b-openshift-default-dockercfg-kwnr4\""
Sep 14 09:57:32.877490 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:32.877452    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk"]
Sep 14 09:57:32.900603 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:57:32.900507    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7690-f9bf0c\" is forbidden: User \"system:node:ip-10-0-132-4.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-4.ec2.internal' and this object" podUID="1dcbc0d5-6f08-4f01-a7c5-7300773fc6c3" pod="opendatahub/test-curl-7690-f9bf0c"
Sep 14 09:57:32.939673 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:32.939645    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:33.019091 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:33.019036    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/770bb961-11e9-43cc-8017-e9b3a041b308-workload-socket\") pod \"e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk\" (UID: \"770bb961-11e9-43cc-8017-e9b3a041b308\") " pod="openshift-ingress/e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk"
Sep 14 09:57:33.019091 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:33.019099    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/770bb961-11e9-43cc-8017-e9b3a041b308-istio-podinfo\") pod \"e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk\" (UID: \"770bb961-11e9-43cc-8017-e9b3a041b308\") " pod="openshift-ingress/e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk"
Sep 14 09:57:33.019359 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:33.019150    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/770bb961-11e9-43cc-8017-e9b3a041b308-credential-socket\") pod \"e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk\" (UID: \"770bb961-11e9-43cc-8017-e9b3a041b308\") " pod="openshift-ingress/e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk"
Sep 14 09:57:33.019359 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:33.019225    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/770bb961-11e9-43cc-8017-e9b3a041b308-istiod-ca-cert\") pod \"e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk\" (UID: \"770bb961-11e9-43cc-8017-e9b3a041b308\") " pod="openshift-ingress/e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk"
Sep 14 09:57:33.019359 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:33.019264    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/770bb961-11e9-43cc-8017-e9b3a041b308-istio-token\") pod \"e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk\" (UID: \"770bb961-11e9-43cc-8017-e9b3a041b308\") " pod="openshift-ingress/e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk"
Sep 14 09:57:33.019359 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:33.019306    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/770bb961-11e9-43cc-8017-e9b3a041b308-istio-envoy\") pod \"e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk\" (UID: \"770bb961-11e9-43cc-8017-e9b3a041b308\") " pod="openshift-ingress/e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk"
Sep 14 09:57:33.019359 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:33.019333    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/770bb961-11e9-43cc-8017-e9b3a041b308-workload-certs\") pod \"e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk\" (UID: \"770bb961-11e9-43cc-8017-e9b3a041b308\") " pod="openshift-ingress/e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk"
Sep 14 09:57:33.019660 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:33.019455    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/770bb961-11e9-43cc-8017-e9b3a041b308-istio-data\") pod \"e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk\" (UID: \"770bb961-11e9-43cc-8017-e9b3a041b308\") " pod="openshift-ingress/e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk"
Sep 14 09:57:33.019660 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:33.019495    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76sdt\" (UniqueName: \"kubernetes.io/projected/770bb961-11e9-43cc-8017-e9b3a041b308-kube-api-access-76sdt\") pod \"e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk\" (UID: \"770bb961-11e9-43cc-8017-e9b3a041b308\") " pod="openshift-ingress/e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk"
Sep 14 09:57:33.081819 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:33.081777    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:33.120227 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:33.120189    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/770bb961-11e9-43cc-8017-e9b3a041b308-istiod-ca-cert\") pod \"e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk\" (UID: \"770bb961-11e9-43cc-8017-e9b3a041b308\") " pod="openshift-ingress/e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk"
Sep 14 09:57:33.120227 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:33.120229    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/770bb961-11e9-43cc-8017-e9b3a041b308-istio-token\") pod \"e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk\" (UID: \"770bb961-11e9-43cc-8017-e9b3a041b308\") " pod="openshift-ingress/e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk"
Sep 14 09:57:33.120475 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:33.120342    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/770bb961-11e9-43cc-8017-e9b3a041b308-istio-envoy\") pod \"e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk\" (UID: \"770bb961-11e9-43cc-8017-e9b3a041b308\") " pod="openshift-ingress/e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk"
Sep 14 09:57:33.120475 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:33.120379    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/770bb961-11e9-43cc-8017-e9b3a041b308-workload-certs\") pod \"e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk\" (UID: \"770bb961-11e9-43cc-8017-e9b3a041b308\") " pod="openshift-ingress/e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk"
Sep 14 09:57:33.120601 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:33.120481    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/770bb961-11e9-43cc-8017-e9b3a041b308-istio-data\") pod \"e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk\" (UID: \"770bb961-11e9-43cc-8017-e9b3a041b308\") " pod="openshift-ingress/e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk"
Sep 14 09:57:33.120601 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:33.120524    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76sdt\" (UniqueName: \"kubernetes.io/projected/770bb961-11e9-43cc-8017-e9b3a041b308-kube-api-access-76sdt\") pod \"e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk\" (UID: \"770bb961-11e9-43cc-8017-e9b3a041b308\") " pod="openshift-ingress/e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk"
Sep 14 09:57:33.120601 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:33.120564    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/770bb961-11e9-43cc-8017-e9b3a041b308-workload-socket\") pod \"e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk\" (UID: \"770bb961-11e9-43cc-8017-e9b3a041b308\") " pod="openshift-ingress/e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk"
Sep 14 09:57:33.120601 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:33.120592    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/770bb961-11e9-43cc-8017-e9b3a041b308-istio-podinfo\") pod \"e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk\" (UID: \"770bb961-11e9-43cc-8017-e9b3a041b308\") " pod="openshift-ingress/e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk"
Sep 14 09:57:33.120803 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:33.120638    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/770bb961-11e9-43cc-8017-e9b3a041b308-credential-socket\") pod \"e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk\" (UID: \"770bb961-11e9-43cc-8017-e9b3a041b308\") " pod="openshift-ingress/e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk"
Sep 14 09:57:33.120871 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:33.120806    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/770bb961-11e9-43cc-8017-e9b3a041b308-workload-certs\") pod \"e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk\" (UID: \"770bb961-11e9-43cc-8017-e9b3a041b308\") " pod="openshift-ingress/e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk"
Sep 14 09:57:33.120924 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:33.120879    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/770bb961-11e9-43cc-8017-e9b3a041b308-istio-data\") pod \"e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk\" (UID: \"770bb961-11e9-43cc-8017-e9b3a041b308\") " pod="openshift-ingress/e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk"
Sep 14 09:57:33.120980 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:33.120935    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/770bb961-11e9-43cc-8017-e9b3a041b308-credential-socket\") pod \"e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk\" (UID: \"770bb961-11e9-43cc-8017-e9b3a041b308\") " pod="openshift-ingress/e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk"
Sep 14 09:57:33.121107 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:33.121079    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/770bb961-11e9-43cc-8017-e9b3a041b308-istiod-ca-cert\") pod \"e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk\" (UID: \"770bb961-11e9-43cc-8017-e9b3a041b308\") " pod="openshift-ingress/e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk"
Sep 14 09:57:33.121235 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:33.121199    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/770bb961-11e9-43cc-8017-e9b3a041b308-workload-socket\") pod \"e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk\" (UID: \"770bb961-11e9-43cc-8017-e9b3a041b308\") " pod="openshift-ingress/e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk"
Sep 14 09:57:33.122723 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:33.122703    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/770bb961-11e9-43cc-8017-e9b3a041b308-istio-envoy\") pod \"e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk\" (UID: \"770bb961-11e9-43cc-8017-e9b3a041b308\") " pod="openshift-ingress/e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk"
Sep 14 09:57:33.123171 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:33.123148    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/770bb961-11e9-43cc-8017-e9b3a041b308-istio-podinfo\") pod \"e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk\" (UID: \"770bb961-11e9-43cc-8017-e9b3a041b308\") " pod="openshift-ingress/e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk"
Sep 14 09:57:33.129108 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:33.129036    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/770bb961-11e9-43cc-8017-e9b3a041b308-istio-token\") pod \"e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk\" (UID: \"770bb961-11e9-43cc-8017-e9b3a041b308\") " pod="openshift-ingress/e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk"
Sep 14 09:57:33.129388 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:33.129333    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76sdt\" (UniqueName: \"kubernetes.io/projected/770bb961-11e9-43cc-8017-e9b3a041b308-kube-api-access-76sdt\") pod \"e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk\" (UID: \"770bb961-11e9-43cc-8017-e9b3a041b308\") " pod="openshift-ingress/e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk"
Sep 14 09:57:33.173199 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:33.173102    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk"
Sep 14 09:57:33.474511 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:33.474474    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk"]
Sep 14 09:57:33.474673 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:57:33.474644    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod770bb961_11e9_43cc_8017_e9b3a041b308.slice/crio-35d29c5c8f5a89df0de4ea5288fbfa5115577cb103ebb369f39d08c4d89ecda4 WatchSource:0}: Error finding container 35d29c5c8f5a89df0de4ea5288fbfa5115577cb103ebb369f39d08c4d89ecda4: Status 404 returned error can't find the container with id 35d29c5c8f5a89df0de4ea5288fbfa5115577cb103ebb369f39d08c4d89ecda4
Sep 14 09:57:33.477953 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:33.477908    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 09:57:33.478095 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:33.478054    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 09:57:33.478161 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:33.478100    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 09:57:33.483452 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:33.483417    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1dcbc0d5-6f08-4f01-a7c5-7300773fc6c3" path="/var/lib/kubelet/pods/1dcbc0d5-6f08-4f01-a7c5-7300773fc6c3/volumes"
Sep 14 09:57:33.675471 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:33.675415    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:33.838738 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:33.838699    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk" event={"ID":"770bb961-11e9-43cc-8017-e9b3a041b308","Type":"ContainerStarted","Data":"7c8a9951b414f2c84ebe788348949ce8df0c56886ead0908f9804ea3943683fd"}
Sep 14 09:57:33.838958 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:33.838755    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk" event={"ID":"770bb961-11e9-43cc-8017-e9b3a041b308","Type":"ContainerStarted","Data":"35d29c5c8f5a89df0de4ea5288fbfa5115577cb103ebb369f39d08c4d89ecda4"}
Sep 14 09:57:33.871056 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:33.870956    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk" podStartSLOduration=1.870936735 podStartE2EDuration="1.870936735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:57:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:57:33.86896981 +0000 UTC m=+842.908644517" watchObservedRunningTime="2026-09-14 09:57:33.870936735 +0000 UTC m=+842.910611356"
Sep 14 09:57:34.174259 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:34.174162    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk"
Sep 14 09:57:34.179871 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:34.179832    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk"
Sep 14 09:57:34.847463 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:34.847426    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk"
Sep 14 09:57:34.849858 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:34.849818    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk"
Sep 14 09:57:35.102406 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:35.102313    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:35.383752 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:35.383652    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:35.499289 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:35.499254    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:36.219236 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:36.219207    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:36.315034 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:36.314966    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:36.416969 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:36.416932    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:37.213209 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:37.213170    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:38.375038 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:38.373741    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:38.476869 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:38.476774    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:38.573934 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:38.573887    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:39.015609 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:39.015544    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:39.114750 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:39.114719    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:39.523602 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:39.523481    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:39.651793 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:39.651662    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:39.768739 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:39.768701    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:39.981451 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:39.981340    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:40.156219 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:40.156177    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:40.264671 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:40.264606    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:40.663929 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:40.663884    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:40.767221 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:40.767186    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:41.494188 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:41.494150    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:42.336959 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:42.336907    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:42.567980 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:42.567932    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:43.415020 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:43.414964    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:44.134157 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:44.134118    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:44.396157 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:44.396057    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:44.583206 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:44.583163    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:44.968350 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:44.968242    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:45.231376 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:45.231292    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-24873d-openshift-default-849786766d-xx9wt"]
Sep 14 09:57:45.246874 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:45.246851    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-24873d-openshift-default-849786766d-xx9wt"
Sep 14 09:57:45.249827 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:45.249794    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-24873d-openshift-default-dockercfg-qmxhw\""
Sep 14 09:57:45.255416 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:45.255385    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-24873d-openshift-default-849786766d-xx9wt"]
Sep 14 09:57:45.260071 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:45.260038    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:45.310906 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:45.310814    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e67fbb48-e200-495b-8af4-590c28adf33d-istio-podinfo\") pod \"e2e-inf-b-24873d-openshift-default-849786766d-xx9wt\" (UID: \"e67fbb48-e200-495b-8af4-590c28adf33d\") " pod="openshift-ingress/e2e-inf-b-24873d-openshift-default-849786766d-xx9wt"
Sep 14 09:57:45.310906 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:45.310862    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e67fbb48-e200-495b-8af4-590c28adf33d-istio-token\") pod \"e2e-inf-b-24873d-openshift-default-849786766d-xx9wt\" (UID: \"e67fbb48-e200-495b-8af4-590c28adf33d\") " pod="openshift-ingress/e2e-inf-b-24873d-openshift-default-849786766d-xx9wt"
Sep 14 09:57:45.311153 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:45.310927    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e67fbb48-e200-495b-8af4-590c28adf33d-istiod-ca-cert\") pod \"e2e-inf-b-24873d-openshift-default-849786766d-xx9wt\" (UID: \"e67fbb48-e200-495b-8af4-590c28adf33d\") " pod="openshift-ingress/e2e-inf-b-24873d-openshift-default-849786766d-xx9wt"
Sep 14 09:57:45.311153 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:45.310965    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e67fbb48-e200-495b-8af4-590c28adf33d-credential-socket\") pod \"e2e-inf-b-24873d-openshift-default-849786766d-xx9wt\" (UID: \"e67fbb48-e200-495b-8af4-590c28adf33d\") " pod="openshift-ingress/e2e-inf-b-24873d-openshift-default-849786766d-xx9wt"
Sep 14 09:57:45.311153 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:45.310988    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e67fbb48-e200-495b-8af4-590c28adf33d-istio-envoy\") pod \"e2e-inf-b-24873d-openshift-default-849786766d-xx9wt\" (UID: \"e67fbb48-e200-495b-8af4-590c28adf33d\") " pod="openshift-ingress/e2e-inf-b-24873d-openshift-default-849786766d-xx9wt"
Sep 14 09:57:45.311153 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:45.311074    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e67fbb48-e200-495b-8af4-590c28adf33d-istio-data\") pod \"e2e-inf-b-24873d-openshift-default-849786766d-xx9wt\" (UID: \"e67fbb48-e200-495b-8af4-590c28adf33d\") " pod="openshift-ingress/e2e-inf-b-24873d-openshift-default-849786766d-xx9wt"
Sep 14 09:57:45.311153 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:45.311105    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e67fbb48-e200-495b-8af4-590c28adf33d-workload-certs\") pod \"e2e-inf-b-24873d-openshift-default-849786766d-xx9wt\" (UID: \"e67fbb48-e200-495b-8af4-590c28adf33d\") " pod="openshift-ingress/e2e-inf-b-24873d-openshift-default-849786766d-xx9wt"
Sep 14 09:57:45.311153 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:45.311132    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h28nx\" (UniqueName: \"kubernetes.io/projected/e67fbb48-e200-495b-8af4-590c28adf33d-kube-api-access-h28nx\") pod \"e2e-inf-b-24873d-openshift-default-849786766d-xx9wt\" (UID: \"e67fbb48-e200-495b-8af4-590c28adf33d\") " pod="openshift-ingress/e2e-inf-b-24873d-openshift-default-849786766d-xx9wt"
Sep 14 09:57:45.311153 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:45.311153    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e67fbb48-e200-495b-8af4-590c28adf33d-workload-socket\") pod \"e2e-inf-b-24873d-openshift-default-849786766d-xx9wt\" (UID: \"e67fbb48-e200-495b-8af4-590c28adf33d\") " pod="openshift-ingress/e2e-inf-b-24873d-openshift-default-849786766d-xx9wt"
Sep 14 09:57:45.358653 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:45.358607    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:45.412946 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:45.412232    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e67fbb48-e200-495b-8af4-590c28adf33d-istio-token\") pod \"e2e-inf-b-24873d-openshift-default-849786766d-xx9wt\" (UID: \"e67fbb48-e200-495b-8af4-590c28adf33d\") " pod="openshift-ingress/e2e-inf-b-24873d-openshift-default-849786766d-xx9wt"
Sep 14 09:57:45.412946 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:45.412313    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e67fbb48-e200-495b-8af4-590c28adf33d-istiod-ca-cert\") pod \"e2e-inf-b-24873d-openshift-default-849786766d-xx9wt\" (UID: \"e67fbb48-e200-495b-8af4-590c28adf33d\") " pod="openshift-ingress/e2e-inf-b-24873d-openshift-default-849786766d-xx9wt"
Sep 14 09:57:45.412946 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:45.412347    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e67fbb48-e200-495b-8af4-590c28adf33d-credential-socket\") pod \"e2e-inf-b-24873d-openshift-default-849786766d-xx9wt\" (UID: \"e67fbb48-e200-495b-8af4-590c28adf33d\") " pod="openshift-ingress/e2e-inf-b-24873d-openshift-default-849786766d-xx9wt"
Sep 14 09:57:45.412946 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:45.412376    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e67fbb48-e200-495b-8af4-590c28adf33d-istio-envoy\") pod \"e2e-inf-b-24873d-openshift-default-849786766d-xx9wt\" (UID: \"e67fbb48-e200-495b-8af4-590c28adf33d\") " pod="openshift-ingress/e2e-inf-b-24873d-openshift-default-849786766d-xx9wt"
Sep 14 09:57:45.412946 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:45.412433    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e67fbb48-e200-495b-8af4-590c28adf33d-istio-data\") pod \"e2e-inf-b-24873d-openshift-default-849786766d-xx9wt\" (UID: \"e67fbb48-e200-495b-8af4-590c28adf33d\") " pod="openshift-ingress/e2e-inf-b-24873d-openshift-default-849786766d-xx9wt"
Sep 14 09:57:45.412946 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:45.412490    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e67fbb48-e200-495b-8af4-590c28adf33d-workload-certs\") pod \"e2e-inf-b-24873d-openshift-default-849786766d-xx9wt\" (UID: \"e67fbb48-e200-495b-8af4-590c28adf33d\") " pod="openshift-ingress/e2e-inf-b-24873d-openshift-default-849786766d-xx9wt"
Sep 14 09:57:45.412946 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:45.412531    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h28nx\" (UniqueName: \"kubernetes.io/projected/e67fbb48-e200-495b-8af4-590c28adf33d-kube-api-access-h28nx\") pod \"e2e-inf-b-24873d-openshift-default-849786766d-xx9wt\" (UID: \"e67fbb48-e200-495b-8af4-590c28adf33d\") " pod="openshift-ingress/e2e-inf-b-24873d-openshift-default-849786766d-xx9wt"
Sep 14 09:57:45.412946 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:45.412571    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e67fbb48-e200-495b-8af4-590c28adf33d-workload-socket\") pod \"e2e-inf-b-24873d-openshift-default-849786766d-xx9wt\" (UID: \"e67fbb48-e200-495b-8af4-590c28adf33d\") " pod="openshift-ingress/e2e-inf-b-24873d-openshift-default-849786766d-xx9wt"
Sep 14 09:57:45.412946 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:45.412651    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e67fbb48-e200-495b-8af4-590c28adf33d-istio-podinfo\") pod \"e2e-inf-b-24873d-openshift-default-849786766d-xx9wt\" (UID: \"e67fbb48-e200-495b-8af4-590c28adf33d\") " pod="openshift-ingress/e2e-inf-b-24873d-openshift-default-849786766d-xx9wt"
Sep 14 09:57:45.413812 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:45.413668    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e67fbb48-e200-495b-8af4-590c28adf33d-workload-certs\") pod \"e2e-inf-b-24873d-openshift-default-849786766d-xx9wt\" (UID: \"e67fbb48-e200-495b-8af4-590c28adf33d\") " pod="openshift-ingress/e2e-inf-b-24873d-openshift-default-849786766d-xx9wt"
Sep 14 09:57:45.413952 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:45.413836    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e67fbb48-e200-495b-8af4-590c28adf33d-credential-socket\") pod \"e2e-inf-b-24873d-openshift-default-849786766d-xx9wt\" (UID: \"e67fbb48-e200-495b-8af4-590c28adf33d\") " pod="openshift-ingress/e2e-inf-b-24873d-openshift-default-849786766d-xx9wt"
Sep 14 09:57:45.414041 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:45.413996    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e67fbb48-e200-495b-8af4-590c28adf33d-workload-socket\") pod \"e2e-inf-b-24873d-openshift-default-849786766d-xx9wt\" (UID: \"e67fbb48-e200-495b-8af4-590c28adf33d\") " pod="openshift-ingress/e2e-inf-b-24873d-openshift-default-849786766d-xx9wt"
Sep 14 09:57:45.414161 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:45.414109    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e67fbb48-e200-495b-8af4-590c28adf33d-istiod-ca-cert\") pod \"e2e-inf-b-24873d-openshift-default-849786766d-xx9wt\" (UID: \"e67fbb48-e200-495b-8af4-590c28adf33d\") " pod="openshift-ingress/e2e-inf-b-24873d-openshift-default-849786766d-xx9wt"
Sep 14 09:57:45.414252 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:45.414215    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e67fbb48-e200-495b-8af4-590c28adf33d-istio-data\") pod \"e2e-inf-b-24873d-openshift-default-849786766d-xx9wt\" (UID: \"e67fbb48-e200-495b-8af4-590c28adf33d\") " pod="openshift-ingress/e2e-inf-b-24873d-openshift-default-849786766d-xx9wt"
Sep 14 09:57:45.416033 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:45.415985    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e67fbb48-e200-495b-8af4-590c28adf33d-istio-podinfo\") pod \"e2e-inf-b-24873d-openshift-default-849786766d-xx9wt\" (UID: \"e67fbb48-e200-495b-8af4-590c28adf33d\") " pod="openshift-ingress/e2e-inf-b-24873d-openshift-default-849786766d-xx9wt"
Sep 14 09:57:45.416033 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:45.416022    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e67fbb48-e200-495b-8af4-590c28adf33d-istio-envoy\") pod \"e2e-inf-b-24873d-openshift-default-849786766d-xx9wt\" (UID: \"e67fbb48-e200-495b-8af4-590c28adf33d\") " pod="openshift-ingress/e2e-inf-b-24873d-openshift-default-849786766d-xx9wt"
Sep 14 09:57:45.421960 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:45.421912    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h28nx\" (UniqueName: \"kubernetes.io/projected/e67fbb48-e200-495b-8af4-590c28adf33d-kube-api-access-h28nx\") pod \"e2e-inf-b-24873d-openshift-default-849786766d-xx9wt\" (UID: \"e67fbb48-e200-495b-8af4-590c28adf33d\") " pod="openshift-ingress/e2e-inf-b-24873d-openshift-default-849786766d-xx9wt"
Sep 14 09:57:45.425396 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:45.424617    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e67fbb48-e200-495b-8af4-590c28adf33d-istio-token\") pod \"e2e-inf-b-24873d-openshift-default-849786766d-xx9wt\" (UID: \"e67fbb48-e200-495b-8af4-590c28adf33d\") " pod="openshift-ingress/e2e-inf-b-24873d-openshift-default-849786766d-xx9wt"
Sep 14 09:57:45.533723 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:45.533668    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:45.560621 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:45.560564    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-24873d-openshift-default-849786766d-xx9wt"
Sep 14 09:57:45.647775 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:45.647712    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:45.753323 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:45.749586    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:45.885503 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:45.885470    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:45.933933 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:45.933888    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-24873d-openshift-default-849786766d-xx9wt"]
Sep 14 09:57:45.972651 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:45.971448    2563 generic.go:356] "Generic (PLEG): container finished" podID="a9d150e6-194e-44be-8a5f-bb826032f664" containerID="d42c378f47c1e6616509d23a6f306ae3a147e0c5e1bae984a4774ac1fe3aa63e" exitCode=0
Sep 14 09:57:45.972651 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:45.971500    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw" event={"ID":"a9d150e6-194e-44be-8a5f-bb826032f664","Type":"ContainerDied","Data":"d42c378f47c1e6616509d23a6f306ae3a147e0c5e1bae984a4774ac1fe3aa63e"}
Sep 14 09:57:45.972651 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:45.972086    2563 scope.go:117] "RemoveContainer" containerID="d42c378f47c1e6616509d23a6f306ae3a147e0c5e1bae984a4774ac1fe3aa63e"
Sep 14 09:57:45.975809 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:45.974420    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 09:57:45.975809 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:45.974535    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 09:57:45.975809 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:45.974587    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 09:57:45.981979 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:45.981381    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 09:57:45.981979 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:45.981471    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 09:57:45.981979 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:45.981516    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 09:57:46.429459 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:46.428626    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:46.981372 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:46.981344    2563 generic.go:356] "Generic (PLEG): container finished" podID="a68cc340-1e5b-4971-96cc-91a62af7d642" containerID="2582027d20843d4e4febcf69ccbb6c0043ba638676372fb2b834cfa4cea62f31" exitCode=0
Sep 14 09:57:46.981941 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:46.981908    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn" event={"ID":"a68cc340-1e5b-4971-96cc-91a62af7d642","Type":"ContainerDied","Data":"2582027d20843d4e4febcf69ccbb6c0043ba638676372fb2b834cfa4cea62f31"}
Sep 14 09:57:46.982613 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:46.982584    2563 scope.go:117] "RemoveContainer" containerID="2582027d20843d4e4febcf69ccbb6c0043ba638676372fb2b834cfa4cea62f31"
Sep 14 09:57:46.984077 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:46.984049    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 09:57:46.984169 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:46.984131    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 09:57:46.984169 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:46.984166    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 09:57:46.984861 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:46.984828    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw" event={"ID":"a9d150e6-194e-44be-8a5f-bb826032f664","Type":"ContainerStarted","Data":"606094d57bf0e58f8d128d7feda6dd7fe2c4919e29a3f387551845738a77ac18"}
Sep 14 09:57:46.987516 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:46.987489    2563 generic.go:356] "Generic (PLEG): container finished" podID="3188c970-5b86-4c03-95cc-12480aa11882" containerID="f7e0b1923c1515b6523444260d8e8f86a71e1c4c8e8a06270c6cbc3fa374ab22" exitCode=0
Sep 14 09:57:46.987578 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:46.987556    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m" event={"ID":"3188c970-5b86-4c03-95cc-12480aa11882","Type":"ContainerDied","Data":"f7e0b1923c1515b6523444260d8e8f86a71e1c4c8e8a06270c6cbc3fa374ab22"}
Sep 14 09:57:46.988169 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:46.988154    2563 scope.go:117] "RemoveContainer" containerID="f7e0b1923c1515b6523444260d8e8f86a71e1c4c8e8a06270c6cbc3fa374ab22"
Sep 14 09:57:46.989565 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:46.989544    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-24873d-openshift-default-849786766d-xx9wt" event={"ID":"e67fbb48-e200-495b-8af4-590c28adf33d","Type":"ContainerStarted","Data":"115cae8af62489681ec015d8bd639be57efee89686e43d7f9ee379018d7c82a3"}
Sep 14 09:57:46.989620 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:46.989589    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-24873d-openshift-default-849786766d-xx9wt" event={"ID":"e67fbb48-e200-495b-8af4-590c28adf33d","Type":"ContainerStarted","Data":"ba5199078dc0609ff936dc1b7adbd3a907d69b1586dba1d6648b1854af751704"}
Sep 14 09:57:46.993738 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:46.993706    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 09:57:46.993806 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:46.993785    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 09:57:46.993842 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:46.993815    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 09:57:47.084433 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:47.084338    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-24873d-openshift-default-849786766d-xx9wt" podStartSLOduration=2.084316037 podStartE2EDuration="2.084316037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:57:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:57:47.08381868 +0000 UTC m=+856.123493301" watchObservedRunningTime="2026-09-14 09:57:47.084316037 +0000 UTC m=+856.123990660"
Sep 14 09:57:47.411762 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:47.411732    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:47.563086 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:47.563053    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-24873d-openshift-default-849786766d-xx9wt"
Sep 14 09:57:47.576392 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:47.576359    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-24873d-openshift-default-849786766d-xx9wt"
Sep 14 09:57:47.632463 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:47.632424    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw"
Sep 14 09:57:47.632810 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:47.632725    2563 patch_prober.go:28] interesting pod/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Sep 14 09:57:47.632985 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:47.632963    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw" podUID="a9d150e6-194e-44be-8a5f-bb826032f664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Sep 14 09:57:48.002366 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:48.002313    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m" event={"ID":"3188c970-5b86-4c03-95cc-12480aa11882","Type":"ContainerStarted","Data":"12e1ac8bfb0d6781d8fe8a63d1df308662a52a39990c18e86f122862fb25fb85"}
Sep 14 09:57:48.012929 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:48.012892    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn" event={"ID":"a68cc340-1e5b-4971-96cc-91a62af7d642","Type":"ContainerStarted","Data":"6422193b01c0dfe2654946eb77ea4b11a6b868b997bb5244e1104c0a6b4f28e6"}
Sep 14 09:57:48.014866 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:48.014834    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-24873d-openshift-default-849786766d-xx9wt"
Sep 14 09:57:48.017555 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:48.017517    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-24873d-openshift-default-849786766d-xx9wt"
Sep 14 09:57:48.419298 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:48.419175    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:48.457167 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:48.457133    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m"
Sep 14 09:57:48.457557 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:48.457468    2563 patch_prober.go:28] interesting pod/e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Sep 14 09:57:48.457700 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:48.457678    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m" podUID="3188c970-5b86-4c03-95cc-12480aa11882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Sep 14 09:57:48.530628 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:48.530592    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn"
Sep 14 09:57:48.531083 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:48.530960    2563 patch_prober.go:28] interesting pod/e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 14 09:57:48.531319 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:48.531293    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn" podUID="a68cc340-1e5b-4971-96cc-91a62af7d642" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 14 09:57:48.629653 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:48.629579    2563 patch_prober.go:28] interesting pod/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Sep 14 09:57:48.629872 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:48.629675    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw" podUID="a9d150e6-194e-44be-8a5f-bb826032f664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Sep 14 09:57:48.958714 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:48.958666    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:49.054853 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:49.054809    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:49.362459 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:49.360840    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:49.456665 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:49.456612    2563 patch_prober.go:28] interesting pod/e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Sep 14 09:57:49.456852 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:49.456693    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m" podUID="3188c970-5b86-4c03-95cc-12480aa11882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Sep 14 09:57:49.530042 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:49.529981    2563 patch_prober.go:28] interesting pod/e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 14 09:57:49.530259 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:49.530077    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn" podUID="a68cc340-1e5b-4971-96cc-91a62af7d642" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 14 09:57:49.630093 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:49.629329    2563 patch_prober.go:28] interesting pod/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Sep 14 09:57:49.630093 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:49.629399    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw" podUID="a9d150e6-194e-44be-8a5f-bb826032f664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Sep 14 09:57:50.457114 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:50.457069    2563 patch_prober.go:28] interesting pod/e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Sep 14 09:57:50.457715 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:50.457682    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m" podUID="3188c970-5b86-4c03-95cc-12480aa11882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Sep 14 09:57:50.530989 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:50.530939    2563 patch_prober.go:28] interesting pod/e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 14 09:57:50.531353 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:50.531296    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn" podUID="a68cc340-1e5b-4971-96cc-91a62af7d642" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 14 09:57:50.608185 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:50.608127    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eed367b-8t8nt"]
Sep 14 09:57:50.618458 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:50.618406    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eed367b-8t8nt"
Sep 14 09:57:50.621324 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:50.621289    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 14 09:57:50.621658 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:50.621627    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gf4xj\""
Sep 14 09:57:50.629567 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:50.629339    2563 patch_prober.go:28] interesting pod/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Sep 14 09:57:50.629567 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:50.629454    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw" podUID="a9d150e6-194e-44be-8a5f-bb826032f664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Sep 14 09:57:50.631954 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:50.630612    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eed367b-8t8nt"]
Sep 14 09:57:50.707033 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:50.706632    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17488da1-cfa8-41d8-bb94-1e6376151cd3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eed367b-8t8nt\" (UID: \"17488da1-cfa8-41d8-bb94-1e6376151cd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eed367b-8t8nt"
Sep 14 09:57:50.711092 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:50.708735    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:50.808028 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:50.807695    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17488da1-cfa8-41d8-bb94-1e6376151cd3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eed367b-8t8nt\" (UID: \"17488da1-cfa8-41d8-bb94-1e6376151cd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eed367b-8t8nt"
Sep 14 09:57:50.812030 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:50.808690    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17488da1-cfa8-41d8-bb94-1e6376151cd3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9eed367b-8t8nt\" (UID: \"17488da1-cfa8-41d8-bb94-1e6376151cd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eed367b-8t8nt"
Sep 14 09:57:50.939216 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:50.939164    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eed367b-8t8nt"
Sep 14 09:57:51.459026 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:51.458489    2563 patch_prober.go:28] interesting pod/e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Sep 14 09:57:51.459026 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:51.458562    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m" podUID="3188c970-5b86-4c03-95cc-12480aa11882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Sep 14 09:57:51.504264 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:57:51.504202    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17488da1_cfa8_41d8_bb94_1e6376151cd3.slice/crio-810330236ea1c1fbcf028b15f5cacdec3f9d267009403c32696179928a1ed4ea WatchSource:0}: Error finding container 810330236ea1c1fbcf028b15f5cacdec3f9d267009403c32696179928a1ed4ea: Status 404 returned error can't find the container with id 810330236ea1c1fbcf028b15f5cacdec3f9d267009403c32696179928a1ed4ea
Sep 14 09:57:51.506081 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:51.506045    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eed367b-8t8nt"]
Sep 14 09:57:51.533566 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:51.533423    2563 patch_prober.go:28] interesting pod/e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 14 09:57:51.533566 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:51.533487    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn" podUID="a68cc340-1e5b-4971-96cc-91a62af7d642" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 14 09:57:51.630028 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:51.629827    2563 patch_prober.go:28] interesting pod/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Sep 14 09:57:51.630028 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:51.629904    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw" podUID="a9d150e6-194e-44be-8a5f-bb826032f664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Sep 14 09:57:51.832746 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:51.832706    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:52.096424 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:52.096321    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eed367b-8t8nt" event={"ID":"17488da1-cfa8-41d8-bb94-1e6376151cd3","Type":"ContainerStarted","Data":"810330236ea1c1fbcf028b15f5cacdec3f9d267009403c32696179928a1ed4ea"}
Sep 14 09:57:52.457186 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:52.457073    2563 patch_prober.go:28] interesting pod/e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Sep 14 09:57:52.457484 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:52.457448    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m" podUID="3188c970-5b86-4c03-95cc-12480aa11882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Sep 14 09:57:52.531931 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:52.531886    2563 patch_prober.go:28] interesting pod/e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 14 09:57:52.532559 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:52.532495    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn" podUID="a68cc340-1e5b-4971-96cc-91a62af7d642" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 14 09:57:52.629361 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:52.629321    2563 patch_prober.go:28] interesting pod/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Sep 14 09:57:52.629665 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:52.629633    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw" podUID="a9d150e6-194e-44be-8a5f-bb826032f664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Sep 14 09:57:52.848560 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:52.848452    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:53.203137 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:53.196812    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:53.294601 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:53.294551    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:53.459143 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:53.456794    2563 patch_prober.go:28] interesting pod/e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Sep 14 09:57:53.459143 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:53.456867    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m" podUID="3188c970-5b86-4c03-95cc-12480aa11882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Sep 14 09:57:53.531187 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:53.530679    2563 patch_prober.go:28] interesting pod/e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 14 09:57:53.531187 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:53.530745    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn" podUID="a68cc340-1e5b-4971-96cc-91a62af7d642" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 14 09:57:53.632960 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:53.632467    2563 patch_prober.go:28] interesting pod/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Sep 14 09:57:53.632960 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:53.632535    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw" podUID="a9d150e6-194e-44be-8a5f-bb826032f664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Sep 14 09:57:53.768792 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:53.767109    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw"
Sep 14 09:57:53.768792 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:53.768067    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m"
Sep 14 09:57:53.769881 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:53.769193    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn"
Sep 14 09:57:54.018243 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:54.018115    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:54.125544 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:54.125382    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:54.220643 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:54.220601    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:54.456935 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:54.456819    2563 patch_prober.go:28] interesting pod/e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Sep 14 09:57:54.456935 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:54.456895    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m" podUID="3188c970-5b86-4c03-95cc-12480aa11882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Sep 14 09:57:54.531468 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:54.531399    2563 patch_prober.go:28] interesting pod/e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 14 09:57:54.531671 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:54.531472    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn" podUID="a68cc340-1e5b-4971-96cc-91a62af7d642" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 14 09:57:54.636735 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:54.636688    2563 patch_prober.go:28] interesting pod/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Sep 14 09:57:54.637269 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:54.636759    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw" podUID="a9d150e6-194e-44be-8a5f-bb826032f664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Sep 14 09:57:54.718034 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:54.717890    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:54.818421 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:54.818340    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:55.258024 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:55.257658    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:55.464311 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:55.464269    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m"
Sep 14 09:57:55.466054 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:55.465958    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m"
Sep 14 09:57:55.638098 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:55.636649    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw"
Sep 14 09:57:55.643741 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:55.642523    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw"
Sep 14 09:57:55.643947 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:55.643746    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn"
Sep 14 09:57:55.646471 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:55.646416    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn"
Sep 14 09:57:56.126883 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:56.126806    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:56.987615 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:56.987569    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:57.161953 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:57.161914    2563 generic.go:356] "Generic (PLEG): container finished" podID="17488da1-cfa8-41d8-bb94-1e6376151cd3" containerID="a42dcf3ce099ff2285c25a2a66947847af7b7823ef5c9bc0e7d6b0dde51e87df" exitCode=0
Sep 14 09:57:57.162129 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:57.162036    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eed367b-8t8nt" event={"ID":"17488da1-cfa8-41d8-bb94-1e6376151cd3","Type":"ContainerDied","Data":"a42dcf3ce099ff2285c25a2a66947847af7b7823ef5c9bc0e7d6b0dde51e87df"}
Sep 14 09:57:57.661806 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:57.661756    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:57.866127 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:57.862518    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:58.169861 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:58.169778    2563 generic.go:356] "Generic (PLEG): container finished" podID="770bb961-11e9-43cc-8017-e9b3a041b308" containerID="7c8a9951b414f2c84ebe788348949ce8df0c56886ead0908f9804ea3943683fd" exitCode=0
Sep 14 09:57:58.170409 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:58.169852    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk" event={"ID":"770bb961-11e9-43cc-8017-e9b3a041b308","Type":"ContainerDied","Data":"7c8a9951b414f2c84ebe788348949ce8df0c56886ead0908f9804ea3943683fd"}
Sep 14 09:57:58.170409 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:58.170367    2563 scope.go:117] "RemoveContainer" containerID="7c8a9951b414f2c84ebe788348949ce8df0c56886ead0908f9804ea3943683fd"
Sep 14 09:57:58.171738 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:58.171716    2563 generic.go:356] "Generic (PLEG): container finished" podID="eae8537b-56e1-4d2a-a43b-de5bb82f78b8" containerID="8afcbeade6f146478a38713532fae180f6a9baa54dfaa533506ce9ad77f83340" exitCode=0
Sep 14 09:57:58.171847 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:58.171740    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn" event={"ID":"eae8537b-56e1-4d2a-a43b-de5bb82f78b8","Type":"ContainerDied","Data":"8afcbeade6f146478a38713532fae180f6a9baa54dfaa533506ce9ad77f83340"}
Sep 14 09:57:58.171847 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:58.171717    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 09:57:58.171932 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:58.171891    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 09:57:58.171932 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:58.171924    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 09:57:58.172333 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:58.172308    2563 scope.go:117] "RemoveContainer" containerID="8afcbeade6f146478a38713532fae180f6a9baa54dfaa533506ce9ad77f83340"
Sep 14 09:57:58.173507 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:58.173480    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 09:57:58.173619 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:58.173553    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 09:57:58.173619 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:58.173591    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 09:57:58.324539 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:58.324499    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eed367b-8t8nt"
Sep 14 09:57:58.384994 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:58.384946    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17488da1-cfa8-41d8-bb94-1e6376151cd3-maas-api-service-ca\") pod \"17488da1-cfa8-41d8-bb94-1e6376151cd3\" (UID: \"17488da1-cfa8-41d8-bb94-1e6376151cd3\") "
Sep 14 09:57:58.385445 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:58.385412    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17488da1-cfa8-41d8-bb94-1e6376151cd3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "17488da1-cfa8-41d8-bb94-1e6376151cd3" (UID: "17488da1-cfa8-41d8-bb94-1e6376151cd3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 09:57:58.486917 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:58.486874    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17488da1-cfa8-41d8-bb94-1e6376151cd3-maas-api-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:57:58.762027 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:58.761915    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:59.180168 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:59.180068    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn" event={"ID":"eae8537b-56e1-4d2a-a43b-de5bb82f78b8","Type":"ContainerStarted","Data":"bd54bf9c93704b7fcef1b04e0b006646544ad125d4a515355c04bf6db9cb205e"}
Sep 14 09:57:59.181697 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:59.181677    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eed367b-8t8nt"
Sep 14 09:57:59.181697 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:59.181690    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eed367b-8t8nt" event={"ID":"17488da1-cfa8-41d8-bb94-1e6376151cd3","Type":"ContainerDied","Data":"810330236ea1c1fbcf028b15f5cacdec3f9d267009403c32696179928a1ed4ea"}
Sep 14 09:57:59.181869 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:59.181732    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="810330236ea1c1fbcf028b15f5cacdec3f9d267009403c32696179928a1ed4ea"
Sep 14 09:57:59.184060 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:59.184028    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk" event={"ID":"770bb961-11e9-43cc-8017-e9b3a041b308","Type":"ContainerStarted","Data":"4967b3e953a2e7d14b6aa7a9fffa9203051b499b536eefe20842551cf98d05a5"}
Sep 14 09:57:59.562687 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:59.562640    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:57:59.702983 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:59.702951    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn"
Sep 14 09:57:59.704137 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:59.704099    2563 patch_prober.go:28] interesting pod/e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 14 09:57:59.704303 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:57:59.704213    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn" podUID="eae8537b-56e1-4d2a-a43b-de5bb82f78b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 14 09:58:00.175134 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:00.175102    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk"
Sep 14 09:58:00.175479 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:00.175451    2563 patch_prober.go:28] interesting pod/e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Sep 14 09:58:00.175546 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:00.175506    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk" podUID="770bb961-11e9-43cc-8017-e9b3a041b308" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Sep 14 09:58:00.618011 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:00.617959    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:58:00.703329 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:00.703286    2563 patch_prober.go:28] interesting pod/e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 14 09:58:00.703547 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:00.703382    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn" podUID="eae8537b-56e1-4d2a-a43b-de5bb82f78b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 14 09:58:01.174608 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:01.174553    2563 patch_prober.go:28] interesting pod/e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Sep 14 09:58:01.174850 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:01.174649    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk" podUID="770bb961-11e9-43cc-8017-e9b3a041b308" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Sep 14 09:58:01.306023 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:01.291607    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:58:01.374935 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:01.374884    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:58:01.454819 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:01.454691    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:58:01.508776 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:01.508700    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eed367b-8t8nt"]
Sep 14 09:58:01.515430 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:01.515382    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9eed367b-8t8nt"]
Sep 14 09:58:01.528973 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:01.528910    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:58:01.661318 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:01.661276    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:58:01.683029 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:01.679386    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk"]
Sep 14 09:58:01.775026 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:01.772157    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk" podUID="770bb961-11e9-43cc-8017-e9b3a041b308" containerName="istio-proxy" containerID="cri-o://4967b3e953a2e7d14b6aa7a9fffa9203051b499b536eefe20842551cf98d05a5" gracePeriod=30
Sep 14 09:58:01.775026 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:01.730406    2563 patch_prober.go:28] interesting pod/e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 14 09:58:01.775026 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:01.772407    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn" podUID="eae8537b-56e1-4d2a-a43b-de5bb82f78b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 14 09:58:01.825197 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:01.825161    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn"
Sep 14 09:58:02.000403 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:02.000357    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:58:02.126368 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:02.126259    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:58:02.215632 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:02.215597    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:58:02.339916 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:02.339839    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:58:02.660789 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:02.660739    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq"]
Sep 14 09:58:02.661674 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:02.661641    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17488da1-cfa8-41d8-bb94-1e6376151cd3" containerName="revoke-keys"
Sep 14 09:58:02.662160 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:02.662140    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="17488da1-cfa8-41d8-bb94-1e6376151cd3" containerName="revoke-keys"
Sep 14 09:58:02.662423 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:02.662405    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="17488da1-cfa8-41d8-bb94-1e6376151cd3" containerName="revoke-keys"
Sep 14 09:58:02.665369 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:02.665338    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq"
Sep 14 09:58:02.676039 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:02.675048    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-2f4712d0-gw-openshift-default-dockercfg-tpt9d\""
Sep 14 09:58:02.754041 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:02.751956    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:58:02.754041 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:02.752157    2563 patch_prober.go:28] interesting pod/e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 14 09:58:02.754041 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:02.752233    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn" podUID="eae8537b-56e1-4d2a-a43b-de5bb82f78b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 14 09:58:02.754041 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:02.752712    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq"]
Sep 14 09:58:02.805193 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:02.802180    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cf814db-8f7c-437c-a8ca-f6bdd70330c5-istiod-ca-cert\") pod \"e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq\" (UID: \"0cf814db-8f7c-437c-a8ca-f6bdd70330c5\") " pod="openshift-ingress/e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq"
Sep 14 09:58:02.805193 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:02.802286    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cf814db-8f7c-437c-a8ca-f6bdd70330c5-istio-envoy\") pod \"e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq\" (UID: \"0cf814db-8f7c-437c-a8ca-f6bdd70330c5\") " pod="openshift-ingress/e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq"
Sep 14 09:58:02.805193 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:02.802332    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cf814db-8f7c-437c-a8ca-f6bdd70330c5-workload-certs\") pod \"e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq\" (UID: \"0cf814db-8f7c-437c-a8ca-f6bdd70330c5\") " pod="openshift-ingress/e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq"
Sep 14 09:58:02.805193 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:02.802383    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cf814db-8f7c-437c-a8ca-f6bdd70330c5-workload-socket\") pod \"e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq\" (UID: \"0cf814db-8f7c-437c-a8ca-f6bdd70330c5\") " pod="openshift-ingress/e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq"
Sep 14 09:58:02.805193 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:02.802434    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cf814db-8f7c-437c-a8ca-f6bdd70330c5-istio-token\") pod \"e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq\" (UID: \"0cf814db-8f7c-437c-a8ca-f6bdd70330c5\") " pod="openshift-ingress/e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq"
Sep 14 09:58:02.805193 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:02.802475    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cf814db-8f7c-437c-a8ca-f6bdd70330c5-credential-socket\") pod \"e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq\" (UID: \"0cf814db-8f7c-437c-a8ca-f6bdd70330c5\") " pod="openshift-ingress/e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq"
Sep 14 09:58:02.805193 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:02.802518    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cf814db-8f7c-437c-a8ca-f6bdd70330c5-istio-data\") pod \"e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq\" (UID: \"0cf814db-8f7c-437c-a8ca-f6bdd70330c5\") " pod="openshift-ingress/e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq"
Sep 14 09:58:02.805193 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:02.802571    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgddn\" (UniqueName: \"kubernetes.io/projected/0cf814db-8f7c-437c-a8ca-f6bdd70330c5-kube-api-access-tgddn\") pod \"e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq\" (UID: \"0cf814db-8f7c-437c-a8ca-f6bdd70330c5\") " pod="openshift-ingress/e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq"
Sep 14 09:58:02.805193 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:02.802630    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cf814db-8f7c-437c-a8ca-f6bdd70330c5-istio-podinfo\") pod \"e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq\" (UID: \"0cf814db-8f7c-437c-a8ca-f6bdd70330c5\") " pod="openshift-ingress/e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq"
Sep 14 09:58:02.904041 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:02.903899    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cf814db-8f7c-437c-a8ca-f6bdd70330c5-istiod-ca-cert\") pod \"e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq\" (UID: \"0cf814db-8f7c-437c-a8ca-f6bdd70330c5\") " pod="openshift-ingress/e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq"
Sep 14 09:58:02.904041 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:02.903991    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cf814db-8f7c-437c-a8ca-f6bdd70330c5-istio-envoy\") pod \"e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq\" (UID: \"0cf814db-8f7c-437c-a8ca-f6bdd70330c5\") " pod="openshift-ingress/e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq"
Sep 14 09:58:02.904041 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:02.904049    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cf814db-8f7c-437c-a8ca-f6bdd70330c5-workload-certs\") pod \"e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq\" (UID: \"0cf814db-8f7c-437c-a8ca-f6bdd70330c5\") " pod="openshift-ingress/e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq"
Sep 14 09:58:02.904395 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:02.904089    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cf814db-8f7c-437c-a8ca-f6bdd70330c5-workload-socket\") pod \"e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq\" (UID: \"0cf814db-8f7c-437c-a8ca-f6bdd70330c5\") " pod="openshift-ingress/e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq"
Sep 14 09:58:02.904395 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:02.904127    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cf814db-8f7c-437c-a8ca-f6bdd70330c5-istio-token\") pod \"e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq\" (UID: \"0cf814db-8f7c-437c-a8ca-f6bdd70330c5\") " pod="openshift-ingress/e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq"
Sep 14 09:58:02.904395 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:02.904174    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cf814db-8f7c-437c-a8ca-f6bdd70330c5-credential-socket\") pod \"e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq\" (UID: \"0cf814db-8f7c-437c-a8ca-f6bdd70330c5\") " pod="openshift-ingress/e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq"
Sep 14 09:58:02.904395 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:02.904207    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cf814db-8f7c-437c-a8ca-f6bdd70330c5-istio-data\") pod \"e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq\" (UID: \"0cf814db-8f7c-437c-a8ca-f6bdd70330c5\") " pod="openshift-ingress/e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq"
Sep 14 09:58:02.904395 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:02.904250    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgddn\" (UniqueName: \"kubernetes.io/projected/0cf814db-8f7c-437c-a8ca-f6bdd70330c5-kube-api-access-tgddn\") pod \"e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq\" (UID: \"0cf814db-8f7c-437c-a8ca-f6bdd70330c5\") " pod="openshift-ingress/e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq"
Sep 14 09:58:02.904395 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:02.904299    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cf814db-8f7c-437c-a8ca-f6bdd70330c5-istio-podinfo\") pod \"e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq\" (UID: \"0cf814db-8f7c-437c-a8ca-f6bdd70330c5\") " pod="openshift-ingress/e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq"
Sep 14 09:58:02.905530 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:02.905498    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cf814db-8f7c-437c-a8ca-f6bdd70330c5-workload-socket\") pod \"e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq\" (UID: \"0cf814db-8f7c-437c-a8ca-f6bdd70330c5\") " pod="openshift-ingress/e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq"
Sep 14 09:58:02.908020 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:02.906181    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cf814db-8f7c-437c-a8ca-f6bdd70330c5-credential-socket\") pod \"e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq\" (UID: \"0cf814db-8f7c-437c-a8ca-f6bdd70330c5\") " pod="openshift-ingress/e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq"
Sep 14 09:58:02.908258 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:02.906377    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cf814db-8f7c-437c-a8ca-f6bdd70330c5-istio-data\") pod \"e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq\" (UID: \"0cf814db-8f7c-437c-a8ca-f6bdd70330c5\") " pod="openshift-ingress/e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq"
Sep 14 09:58:02.908371 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:02.907347    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cf814db-8f7c-437c-a8ca-f6bdd70330c5-istiod-ca-cert\") pod \"e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq\" (UID: \"0cf814db-8f7c-437c-a8ca-f6bdd70330c5\") " pod="openshift-ingress/e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq"
Sep 14 09:58:02.908473 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:02.907852    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cf814db-8f7c-437c-a8ca-f6bdd70330c5-istio-podinfo\") pod \"e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq\" (UID: \"0cf814db-8f7c-437c-a8ca-f6bdd70330c5\") " pod="openshift-ingress/e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq"
Sep 14 09:58:02.908566 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:02.907943    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cf814db-8f7c-437c-a8ca-f6bdd70330c5-workload-certs\") pod \"e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq\" (UID: \"0cf814db-8f7c-437c-a8ca-f6bdd70330c5\") " pod="openshift-ingress/e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq"
Sep 14 09:58:02.912911 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:02.912414    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cf814db-8f7c-437c-a8ca-f6bdd70330c5-istio-envoy\") pod \"e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq\" (UID: \"0cf814db-8f7c-437c-a8ca-f6bdd70330c5\") " pod="openshift-ingress/e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq"
Sep 14 09:58:02.916055 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:02.915813    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cf814db-8f7c-437c-a8ca-f6bdd70330c5-istio-token\") pod \"e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq\" (UID: \"0cf814db-8f7c-437c-a8ca-f6bdd70330c5\") " pod="openshift-ingress/e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq"
Sep 14 09:58:02.916229 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:02.916086    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgddn\" (UniqueName: \"kubernetes.io/projected/0cf814db-8f7c-437c-a8ca-f6bdd70330c5-kube-api-access-tgddn\") pod \"e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq\" (UID: \"0cf814db-8f7c-437c-a8ca-f6bdd70330c5\") " pod="openshift-ingress/e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq"
Sep 14 09:58:02.933496 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:02.933451    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:58:03.116112 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:03.116067    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq"
Sep 14 09:58:03.134608 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:03.134528    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:58:03.494034 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:03.493612    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17488da1-cfa8-41d8-bb94-1e6376151cd3" path="/var/lib/kubelet/pods/17488da1-cfa8-41d8-bb94-1e6376151cd3/volumes"
Sep 14 09:58:03.540029 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:03.539284    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq"]
Sep 14 09:58:03.570029 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:03.568789    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 09:58:03.570029 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:03.568925    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 09:58:03.570029 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:03.569017    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 09:58:03.703137 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:03.703095    2563 patch_prober.go:28] interesting pod/e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 14 09:58:03.703619 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:03.703161    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn" podUID="eae8537b-56e1-4d2a-a43b-de5bb82f78b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 14 09:58:03.704878 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:03.704838    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:58:04.225906 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:04.224803    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq" event={"ID":"0cf814db-8f7c-437c-a8ca-f6bdd70330c5","Type":"ContainerStarted","Data":"2596ac840736ba3bd2896a6a839cca4b5032094e98d5660ae13a2c546048e143"}
Sep 14 09:58:04.225906 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:04.224878    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq" event={"ID":"0cf814db-8f7c-437c-a8ca-f6bdd70330c5","Type":"ContainerStarted","Data":"e83257e092e41ce5f759d07519bd6c65360f63975adc8ace0efb81190e2504b0"}
Sep 14 09:58:04.252021 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:04.249447    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq" podStartSLOduration=2.249422727 podStartE2EDuration="2.249422727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:58:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:58:04.247728945 +0000 UTC m=+873.287403568" watchObservedRunningTime="2026-09-14 09:58:04.249422727 +0000 UTC m=+873.289097349"
Sep 14 09:58:04.489177 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:04.489127    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:58:04.703283 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:04.702967    2563 patch_prober.go:28] interesting pod/e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 14 09:58:04.703283 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:04.703051    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn" podUID="eae8537b-56e1-4d2a-a43b-de5bb82f78b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 14 09:58:04.847770 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:04.847664    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk"
Sep 14 09:58:05.117438 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:05.117334    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq"
Sep 14 09:58:05.124224 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:05.124193    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq"
Sep 14 09:58:05.235614 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:05.235574    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq"
Sep 14 09:58:05.240667 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:05.240625    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq"
Sep 14 09:58:05.575961 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:05.575920    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:58:05.707544 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:05.707508    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn"
Sep 14 09:58:05.708897 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:05.708869    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn"
Sep 14 09:58:05.751398 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:05.751346    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:58:05.845407 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:05.845309    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:58:06.729463 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:06.729355    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:58:07.506420 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:07.506384    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:58:07.605662 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:07.605626    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:58:07.705220 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:07.705185    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:58:08.024817 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:08.024783    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:58:08.139733 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:08.139698    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:58:08.596727 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:08.596686    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:58:08.792738 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:08.792706    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:58:09.344424 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:09.344377    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:58:10.268700 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:10.268665    2563 generic.go:356] "Generic (PLEG): container finished" podID="e67fbb48-e200-495b-8af4-590c28adf33d" containerID="115cae8af62489681ec015d8bd639be57efee89686e43d7f9ee379018d7c82a3" exitCode=0
Sep 14 09:58:10.268926 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:10.268736    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-24873d-openshift-default-849786766d-xx9wt" event={"ID":"e67fbb48-e200-495b-8af4-590c28adf33d","Type":"ContainerDied","Data":"115cae8af62489681ec015d8bd639be57efee89686e43d7f9ee379018d7c82a3"}
Sep 14 09:58:10.269327 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:10.269306    2563 scope.go:117] "RemoveContainer" containerID="115cae8af62489681ec015d8bd639be57efee89686e43d7f9ee379018d7c82a3"
Sep 14 09:58:10.270681 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:10.270647    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 09:58:10.270789 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:10.270722    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 09:58:10.270789 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:10.270757    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 09:58:10.419035 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:10.418969    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:58:10.767627 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:10.767225    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk"
Sep 14 09:58:10.899957 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:10.899855    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/770bb961-11e9-43cc-8017-e9b3a041b308-istio-envoy\") pod \"770bb961-11e9-43cc-8017-e9b3a041b308\" (UID: \"770bb961-11e9-43cc-8017-e9b3a041b308\") "
Sep 14 09:58:10.899957 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:10.899921    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/770bb961-11e9-43cc-8017-e9b3a041b308-istio-data\") pod \"770bb961-11e9-43cc-8017-e9b3a041b308\" (UID: \"770bb961-11e9-43cc-8017-e9b3a041b308\") "
Sep 14 09:58:10.899957 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:10.899954    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/770bb961-11e9-43cc-8017-e9b3a041b308-workload-socket\") pod \"770bb961-11e9-43cc-8017-e9b3a041b308\" (UID: \"770bb961-11e9-43cc-8017-e9b3a041b308\") "
Sep 14 09:58:10.900313 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:10.899975    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/770bb961-11e9-43cc-8017-e9b3a041b308-istio-podinfo\") pod \"770bb961-11e9-43cc-8017-e9b3a041b308\" (UID: \"770bb961-11e9-43cc-8017-e9b3a041b308\") "
Sep 14 09:58:10.900313 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:10.900045    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-76sdt\" (UniqueName: \"kubernetes.io/projected/770bb961-11e9-43cc-8017-e9b3a041b308-kube-api-access-76sdt\") pod \"770bb961-11e9-43cc-8017-e9b3a041b308\" (UID: \"770bb961-11e9-43cc-8017-e9b3a041b308\") "
Sep 14 09:58:10.900313 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:10.900069    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/770bb961-11e9-43cc-8017-e9b3a041b308-istiod-ca-cert\") pod \"770bb961-11e9-43cc-8017-e9b3a041b308\" (UID: \"770bb961-11e9-43cc-8017-e9b3a041b308\") "
Sep 14 09:58:10.900313 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:10.900096    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/770bb961-11e9-43cc-8017-e9b3a041b308-istio-token\") pod \"770bb961-11e9-43cc-8017-e9b3a041b308\" (UID: \"770bb961-11e9-43cc-8017-e9b3a041b308\") "
Sep 14 09:58:10.900313 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:10.900122    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/770bb961-11e9-43cc-8017-e9b3a041b308-workload-certs\") pod \"770bb961-11e9-43cc-8017-e9b3a041b308\" (UID: \"770bb961-11e9-43cc-8017-e9b3a041b308\") "
Sep 14 09:58:10.900313 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:10.900235    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/770bb961-11e9-43cc-8017-e9b3a041b308-credential-socket\") pod \"770bb961-11e9-43cc-8017-e9b3a041b308\" (UID: \"770bb961-11e9-43cc-8017-e9b3a041b308\") "
Sep 14 09:58:10.900313 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:10.900282    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/770bb961-11e9-43cc-8017-e9b3a041b308-istio-data" (OuterVolumeSpecName: "istio-data") pod "770bb961-11e9-43cc-8017-e9b3a041b308" (UID: "770bb961-11e9-43cc-8017-e9b3a041b308"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 09:58:10.901090 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:10.900315    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/770bb961-11e9-43cc-8017-e9b3a041b308-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "770bb961-11e9-43cc-8017-e9b3a041b308" (UID: "770bb961-11e9-43cc-8017-e9b3a041b308"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 09:58:10.901090 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:10.900480    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/770bb961-11e9-43cc-8017-e9b3a041b308-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "770bb961-11e9-43cc-8017-e9b3a041b308" (UID: "770bb961-11e9-43cc-8017-e9b3a041b308"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 09:58:10.901090 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:10.900520    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/770bb961-11e9-43cc-8017-e9b3a041b308-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "770bb961-11e9-43cc-8017-e9b3a041b308" (UID: "770bb961-11e9-43cc-8017-e9b3a041b308"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 09:58:10.901090 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:10.900620    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/770bb961-11e9-43cc-8017-e9b3a041b308-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:58:10.901090 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:10.900638    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/770bb961-11e9-43cc-8017-e9b3a041b308-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:58:10.901090 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:10.900651    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/770bb961-11e9-43cc-8017-e9b3a041b308-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:58:10.901090 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:10.900665    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/770bb961-11e9-43cc-8017-e9b3a041b308-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:58:10.901090 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:10.900667    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/770bb961-11e9-43cc-8017-e9b3a041b308-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "770bb961-11e9-43cc-8017-e9b3a041b308" (UID: "770bb961-11e9-43cc-8017-e9b3a041b308"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 09:58:10.902466 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:10.902428    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/770bb961-11e9-43cc-8017-e9b3a041b308-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "770bb961-11e9-43cc-8017-e9b3a041b308" (UID: "770bb961-11e9-43cc-8017-e9b3a041b308"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 09:58:10.902594 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:10.902547    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/770bb961-11e9-43cc-8017-e9b3a041b308-istio-token" (OuterVolumeSpecName: "istio-token") pod "770bb961-11e9-43cc-8017-e9b3a041b308" (UID: "770bb961-11e9-43cc-8017-e9b3a041b308"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 09:58:10.902594 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:10.902551    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/770bb961-11e9-43cc-8017-e9b3a041b308-kube-api-access-76sdt" (OuterVolumeSpecName: "kube-api-access-76sdt") pod "770bb961-11e9-43cc-8017-e9b3a041b308" (UID: "770bb961-11e9-43cc-8017-e9b3a041b308"). InnerVolumeSpecName "kube-api-access-76sdt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 09:58:10.902680 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:10.902600    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/770bb961-11e9-43cc-8017-e9b3a041b308-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "770bb961-11e9-43cc-8017-e9b3a041b308" (UID: "770bb961-11e9-43cc-8017-e9b3a041b308"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 09:58:11.001749 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:11.001705    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/770bb961-11e9-43cc-8017-e9b3a041b308-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:58:11.001749 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:11.001741    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76sdt\" (UniqueName: \"kubernetes.io/projected/770bb961-11e9-43cc-8017-e9b3a041b308-kube-api-access-76sdt\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:58:11.001749 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:11.001751    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/770bb961-11e9-43cc-8017-e9b3a041b308-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:58:11.001749 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:11.001760    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/770bb961-11e9-43cc-8017-e9b3a041b308-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:58:11.002062 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:11.001769    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/770bb961-11e9-43cc-8017-e9b3a041b308-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:58:11.277572 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:11.277537    2563 generic.go:356] "Generic (PLEG): container finished" podID="770bb961-11e9-43cc-8017-e9b3a041b308" containerID="4967b3e953a2e7d14b6aa7a9fffa9203051b499b536eefe20842551cf98d05a5" exitCode=0
Sep 14 09:58:11.277784 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:11.277586    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk"
Sep 14 09:58:11.277784 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:11.277623    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk" event={"ID":"770bb961-11e9-43cc-8017-e9b3a041b308","Type":"ContainerDied","Data":"4967b3e953a2e7d14b6aa7a9fffa9203051b499b536eefe20842551cf98d05a5"}
Sep 14 09:58:11.277784 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:11.277682    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk" event={"ID":"770bb961-11e9-43cc-8017-e9b3a041b308","Type":"ContainerDied","Data":"35d29c5c8f5a89df0de4ea5288fbfa5115577cb103ebb369f39d08c4d89ecda4"}
Sep 14 09:58:11.277784 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:11.277706    2563 scope.go:117] "RemoveContainer" containerID="4967b3e953a2e7d14b6aa7a9fffa9203051b499b536eefe20842551cf98d05a5"
Sep 14 09:58:11.280202 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:11.280173    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-24873d-openshift-default-849786766d-xx9wt" event={"ID":"e67fbb48-e200-495b-8af4-590c28adf33d","Type":"ContainerStarted","Data":"34db4e6dcb113290aea30e7b7ce50568a5f9b2f600612d4dd5cdab3e0b5ff879"}
Sep 14 09:58:11.294058 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:11.294025    2563 scope.go:117] "RemoveContainer" containerID="7c8a9951b414f2c84ebe788348949ce8df0c56886ead0908f9804ea3943683fd"
Sep 14 09:58:11.303760 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:11.303726    2563 scope.go:117] "RemoveContainer" containerID="4967b3e953a2e7d14b6aa7a9fffa9203051b499b536eefe20842551cf98d05a5"
Sep 14 09:58:11.304166 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:58:11.304133    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4967b3e953a2e7d14b6aa7a9fffa9203051b499b536eefe20842551cf98d05a5\": container with ID starting with 4967b3e953a2e7d14b6aa7a9fffa9203051b499b536eefe20842551cf98d05a5 not found: ID does not exist" containerID="4967b3e953a2e7d14b6aa7a9fffa9203051b499b536eefe20842551cf98d05a5"
Sep 14 09:58:11.304252 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:11.304186    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4967b3e953a2e7d14b6aa7a9fffa9203051b499b536eefe20842551cf98d05a5"} err="failed to get container status \"4967b3e953a2e7d14b6aa7a9fffa9203051b499b536eefe20842551cf98d05a5\": rpc error: code = NotFound desc = could not find container \"4967b3e953a2e7d14b6aa7a9fffa9203051b499b536eefe20842551cf98d05a5\": container with ID starting with 4967b3e953a2e7d14b6aa7a9fffa9203051b499b536eefe20842551cf98d05a5 not found: ID does not exist"
Sep 14 09:58:11.304252 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:11.304207    2563 scope.go:117] "RemoveContainer" containerID="7c8a9951b414f2c84ebe788348949ce8df0c56886ead0908f9804ea3943683fd"
Sep 14 09:58:11.304516 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:58:11.304496    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c8a9951b414f2c84ebe788348949ce8df0c56886ead0908f9804ea3943683fd\": container with ID starting with 7c8a9951b414f2c84ebe788348949ce8df0c56886ead0908f9804ea3943683fd not found: ID does not exist" containerID="7c8a9951b414f2c84ebe788348949ce8df0c56886ead0908f9804ea3943683fd"
Sep 14 09:58:11.304624 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:11.304526    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c8a9951b414f2c84ebe788348949ce8df0c56886ead0908f9804ea3943683fd"} err="failed to get container status \"7c8a9951b414f2c84ebe788348949ce8df0c56886ead0908f9804ea3943683fd\": rpc error: code = NotFound desc = could not find container \"7c8a9951b414f2c84ebe788348949ce8df0c56886ead0908f9804ea3943683fd\": container with ID starting with 7c8a9951b414f2c84ebe788348949ce8df0c56886ead0908f9804ea3943683fd not found: ID does not exist"
Sep 14 09:58:11.345900 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:11.345848    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk"]
Sep 14 09:58:11.349556 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:11.349520    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-9eed367b-openshift-default-774bb9fdb6-rmfdk"]
Sep 14 09:58:11.482479 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:11.482442    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="770bb961-11e9-43cc-8017-e9b3a041b308" path="/var/lib/kubelet/pods/770bb961-11e9-43cc-8017-e9b3a041b308/volumes"
Sep 14 09:58:11.504791 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:11.504721    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:58:11.561594 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:11.561496    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-24873d-openshift-default-849786766d-xx9wt"
Sep 14 09:58:11.561786 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:11.561761    2563 patch_prober.go:28] interesting pod/e2e-inf-b-24873d-openshift-default-849786766d-xx9wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Sep 14 09:58:11.561845 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:11.561812    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-24873d-openshift-default-849786766d-xx9wt" podUID="e67fbb48-e200-495b-8af4-590c28adf33d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Sep 14 09:58:12.353047 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:12.352980    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:58:12.562232 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:12.562186    2563 patch_prober.go:28] interesting pod/e2e-inf-b-24873d-openshift-default-849786766d-xx9wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Sep 14 09:58:12.562666 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:12.562264    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-24873d-openshift-default-849786766d-xx9wt" podUID="e67fbb48-e200-495b-8af4-590c28adf33d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Sep 14 09:58:13.270305 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:13.270263    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:58:13.561882 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:13.561785    2563 patch_prober.go:28] interesting pod/e2e-inf-b-24873d-openshift-default-849786766d-xx9wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Sep 14 09:58:13.561882 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:13.561852    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-24873d-openshift-default-849786766d-xx9wt" podUID="e67fbb48-e200-495b-8af4-590c28adf33d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Sep 14 09:58:14.562266 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:14.562214    2563 patch_prober.go:28] interesting pod/e2e-inf-b-24873d-openshift-default-849786766d-xx9wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Sep 14 09:58:14.562767 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:14.562287    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-24873d-openshift-default-849786766d-xx9wt" podUID="e67fbb48-e200-495b-8af4-590c28adf33d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Sep 14 09:58:15.171882 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:15.171843    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:58:15.562328 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:15.562296    2563 patch_prober.go:28] interesting pod/e2e-inf-b-24873d-openshift-default-849786766d-xx9wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Sep 14 09:58:15.562846 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:15.562806    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-24873d-openshift-default-849786766d-xx9wt" podUID="e67fbb48-e200-495b-8af4-590c28adf33d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Sep 14 09:58:16.562357 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:16.562307    2563 patch_prober.go:28] interesting pod/e2e-inf-b-24873d-openshift-default-849786766d-xx9wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Sep 14 09:58:16.562856 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:16.562397    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-24873d-openshift-default-849786766d-xx9wt" podUID="e67fbb48-e200-495b-8af4-590c28adf33d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Sep 14 09:58:17.566432 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:17.566398    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-24873d-openshift-default-849786766d-xx9wt"
Sep 14 09:58:17.566909 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:17.566785    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-24873d-openshift-default-849786766d-xx9wt"
Sep 14 09:58:17.567717 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:17.567698    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-24873d-openshift-default-849786766d-xx9wt"
Sep 14 09:58:25.381032 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:25.380979    2563 generic.go:356] "Generic (PLEG): container finished" podID="0cf814db-8f7c-437c-a8ca-f6bdd70330c5" containerID="2596ac840736ba3bd2896a6a839cca4b5032094e98d5660ae13a2c546048e143" exitCode=0
Sep 14 09:58:25.381574 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:25.381061    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq" event={"ID":"0cf814db-8f7c-437c-a8ca-f6bdd70330c5","Type":"ContainerDied","Data":"2596ac840736ba3bd2896a6a839cca4b5032094e98d5660ae13a2c546048e143"}
Sep 14 09:58:25.381670 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:25.381650    2563 scope.go:117] "RemoveContainer" containerID="2596ac840736ba3bd2896a6a839cca4b5032094e98d5660ae13a2c546048e143"
Sep 14 09:58:25.383184 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:25.383148    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 09:58:25.383292 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:25.383237    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 09:58:25.383292 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:25.383277    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 09:58:26.388810 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:26.388764    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq" event={"ID":"0cf814db-8f7c-437c-a8ca-f6bdd70330c5","Type":"ContainerStarted","Data":"9af890ecf7b5a47095b7c0331fc783227937554365348c3f1cd274b8cec3aa3b"}
Sep 14 09:58:27.116753 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:27.116712    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq"
Sep 14 09:58:27.116996 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:27.116942    2563 patch_prober.go:28] interesting pod/e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Sep 14 09:58:27.117106 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:27.117052    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq" podUID="0cf814db-8f7c-437c-a8ca-f6bdd70330c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Sep 14 09:58:28.117146 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:28.117106    2563 patch_prober.go:28] interesting pod/e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Sep 14 09:58:28.117573 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:28.117175    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq" podUID="0cf814db-8f7c-437c-a8ca-f6bdd70330c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Sep 14 09:58:28.508073 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:28.508031    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:58:29.116917 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:29.116877    2563 patch_prober.go:28] interesting pod/e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Sep 14 09:58:29.117158 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:29.116956    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq" podUID="0cf814db-8f7c-437c-a8ca-f6bdd70330c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Sep 14 09:58:29.967238 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:29.967192    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:58:30.117081 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:30.117041    2563 patch_prober.go:28] interesting pod/e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Sep 14 09:58:30.117276 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:30.117106    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq" podUID="0cf814db-8f7c-437c-a8ca-f6bdd70330c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Sep 14 09:58:30.692071 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:30.692023    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2f4712d0-qqsz8"]
Sep 14 09:58:30.692462 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:30.692446    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="770bb961-11e9-43cc-8017-e9b3a041b308" containerName="istio-proxy"
Sep 14 09:58:30.692521 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:30.692463    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="770bb961-11e9-43cc-8017-e9b3a041b308" containerName="istio-proxy"
Sep 14 09:58:30.692521 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:30.692476    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="770bb961-11e9-43cc-8017-e9b3a041b308" containerName="istio-proxy"
Sep 14 09:58:30.692521 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:30.692482    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="770bb961-11e9-43cc-8017-e9b3a041b308" containerName="istio-proxy"
Sep 14 09:58:30.692614 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:30.692534    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="770bb961-11e9-43cc-8017-e9b3a041b308" containerName="istio-proxy"
Sep 14 09:58:30.694807 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:30.694780    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2f4712d0-qqsz8"
Sep 14 09:58:30.698362 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:30.698334    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gf4xj\""
Sep 14 09:58:30.698620 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:30.698602    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 14 09:58:30.710369 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:30.710333    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2f4712d0-qqsz8"]
Sep 14 09:58:30.718926 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:30.718122    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b50df59-7d3b-4f02-a693-f32429f4911c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2f4712d0-qqsz8\" (UID: \"5b50df59-7d3b-4f02-a693-f32429f4911c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2f4712d0-qqsz8"
Sep 14 09:58:30.818778 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:30.818732    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b50df59-7d3b-4f02-a693-f32429f4911c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2f4712d0-qqsz8\" (UID: \"5b50df59-7d3b-4f02-a693-f32429f4911c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2f4712d0-qqsz8"
Sep 14 09:58:30.819433 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:30.819409    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b50df59-7d3b-4f02-a693-f32429f4911c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2f4712d0-qqsz8\" (UID: \"5b50df59-7d3b-4f02-a693-f32429f4911c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2f4712d0-qqsz8"
Sep 14 09:58:31.005791 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:31.005749    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2f4712d0-qqsz8"
Sep 14 09:58:31.096626 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:31.096532    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:58:31.116687 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:31.116643    2563 patch_prober.go:28] interesting pod/e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Sep 14 09:58:31.116873 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:31.116722    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq" podUID="0cf814db-8f7c-437c-a8ca-f6bdd70330c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Sep 14 09:58:31.310553 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:31.310519    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2f4712d0-qqsz8"]
Sep 14 09:58:31.311068 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:58:31.311038    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b50df59_7d3b_4f02_a693_f32429f4911c.slice/crio-53c129fa7b85b470a5bb509e5b138fd26497c0aaafd53cab922b4c88f83ca0aa WatchSource:0}: Error finding container 53c129fa7b85b470a5bb509e5b138fd26497c0aaafd53cab922b4c88f83ca0aa: Status 404 returned error can't find the container with id 53c129fa7b85b470a5bb509e5b138fd26497c0aaafd53cab922b4c88f83ca0aa
Sep 14 09:58:31.437657 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:31.437337    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2f4712d0-qqsz8" event={"ID":"5b50df59-7d3b-4f02-a693-f32429f4911c","Type":"ContainerStarted","Data":"53c129fa7b85b470a5bb509e5b138fd26497c0aaafd53cab922b4c88f83ca0aa"}
Sep 14 09:58:31.989018 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:31.988970    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:58:32.121155 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:32.121123    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq"
Sep 14 09:58:32.121559 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:32.121508    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq"
Sep 14 09:58:32.122295 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:32.122275    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq"
Sep 14 09:58:32.445504 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:32.445413    2563 generic.go:356] "Generic (PLEG): container finished" podID="5b50df59-7d3b-4f02-a693-f32429f4911c" containerID="ec15fb459157c21ab16242fd71d326a98dc301971b7c7e09b3aee9f284c297e0" exitCode=0
Sep 14 09:58:32.445661 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:32.445498    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2f4712d0-qqsz8" event={"ID":"5b50df59-7d3b-4f02-a693-f32429f4911c","Type":"ContainerDied","Data":"ec15fb459157c21ab16242fd71d326a98dc301971b7c7e09b3aee9f284c297e0"}
Sep 14 09:58:32.929175 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:32.929139    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:58:33.256101 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:33.256044    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:58:33.356228 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:33.356193    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:58:33.594070 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:33.594039    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2f4712d0-qqsz8"
Sep 14 09:58:33.667121 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:33.667080    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b50df59-7d3b-4f02-a693-f32429f4911c-maas-api-service-ca\") pod \"5b50df59-7d3b-4f02-a693-f32429f4911c\" (UID: \"5b50df59-7d3b-4f02-a693-f32429f4911c\") "
Sep 14 09:58:33.667476 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:33.667451    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b50df59-7d3b-4f02-a693-f32429f4911c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5b50df59-7d3b-4f02-a693-f32429f4911c" (UID: "5b50df59-7d3b-4f02-a693-f32429f4911c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 09:58:33.768136 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:33.768040    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b50df59-7d3b-4f02-a693-f32429f4911c-maas-api-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:58:33.798821 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:33.798785    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:58:34.460601 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:34.460564    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2f4712d0-qqsz8" event={"ID":"5b50df59-7d3b-4f02-a693-f32429f4911c","Type":"ContainerDied","Data":"53c129fa7b85b470a5bb509e5b138fd26497c0aaafd53cab922b4c88f83ca0aa"}
Sep 14 09:58:34.460601 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:34.460584    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2f4712d0-qqsz8"
Sep 14 09:58:34.461079 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:34.460617    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53c129fa7b85b470a5bb509e5b138fd26497c0aaafd53cab922b4c88f83ca0aa"
Sep 14 09:58:34.606484 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:34.606444    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:58:35.611390 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:35.611355    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:58:36.394744 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:36.394669    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:58:37.080880 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:37.080841    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:58:37.154600 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:37.154557    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:58:37.229080 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:37.229047    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:58:37.323995 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:37.323928    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2f4712d0-qqsz8"]
Sep 14 09:58:37.324247 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:37.324216    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:58:37.326713 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:37.326678    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2f4712d0-qqsz8"]
Sep 14 09:58:37.405034 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:37.404911    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:58:37.456298 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:37.456261    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq"]
Sep 14 09:58:37.456899 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:37.456865    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq" podUID="0cf814db-8f7c-437c-a8ca-f6bdd70330c5" containerName="istio-proxy" containerID="cri-o://9af890ecf7b5a47095b7c0331fc783227937554365348c3f1cd274b8cec3aa3b" gracePeriod=30
Sep 14 09:58:37.484483 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:37.484446    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b50df59-7d3b-4f02-a693-f32429f4911c" path="/var/lib/kubelet/pods/5b50df59-7d3b-4f02-a693-f32429f4911c/volumes"
Sep 14 09:58:37.513341 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:37.513303    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:58:37.830478 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:37.830437    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:58:37.938306 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:37.938270    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:58:38.216394 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:38.216293    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl"]
Sep 14 09:58:38.216763 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:38.216706    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b50df59-7d3b-4f02-a693-f32429f4911c" containerName="revoke-keys"
Sep 14 09:58:38.216763 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:38.216721    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b50df59-7d3b-4f02-a693-f32429f4911c" containerName="revoke-keys"
Sep 14 09:58:38.216835 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:38.216802    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b50df59-7d3b-4f02-a693-f32429f4911c" containerName="revoke-keys"
Sep 14 09:58:38.216835 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:38.216810    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="770bb961-11e9-43cc-8017-e9b3a041b308" containerName="istio-proxy"
Sep 14 09:58:38.218995 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:38.218973    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl"
Sep 14 09:58:38.221236 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:38.221213    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-e7fe0e18-openshift-default-dockercfg-7zxc8\""
Sep 14 09:58:38.233412 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:38.233377    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl"]
Sep 14 09:58:38.248359 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:38.248325    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:58:38.318398 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:38.318354    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3962933b-4378-4941-8983-d4ca1b616be1-istio-podinfo\") pod \"e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl\" (UID: \"3962933b-4378-4941-8983-d4ca1b616be1\") " pod="openshift-ingress/e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl"
Sep 14 09:58:38.318610 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:38.318407    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3962933b-4378-4941-8983-d4ca1b616be1-istio-data\") pod \"e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl\" (UID: \"3962933b-4378-4941-8983-d4ca1b616be1\") " pod="openshift-ingress/e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl"
Sep 14 09:58:38.318610 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:38.318445    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3962933b-4378-4941-8983-d4ca1b616be1-workload-socket\") pod \"e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl\" (UID: \"3962933b-4378-4941-8983-d4ca1b616be1\") " pod="openshift-ingress/e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl"
Sep 14 09:58:38.318610 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:38.318476    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3962933b-4378-4941-8983-d4ca1b616be1-workload-certs\") pod \"e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl\" (UID: \"3962933b-4378-4941-8983-d4ca1b616be1\") " pod="openshift-ingress/e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl"
Sep 14 09:58:38.318610 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:38.318535    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3962933b-4378-4941-8983-d4ca1b616be1-istiod-ca-cert\") pod \"e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl\" (UID: \"3962933b-4378-4941-8983-d4ca1b616be1\") " pod="openshift-ingress/e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl"
Sep 14 09:58:38.318839 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:38.318637    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnhz8\" (UniqueName: \"kubernetes.io/projected/3962933b-4378-4941-8983-d4ca1b616be1-kube-api-access-lnhz8\") pod \"e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl\" (UID: \"3962933b-4378-4941-8983-d4ca1b616be1\") " pod="openshift-ingress/e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl"
Sep 14 09:58:38.318839 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:38.318669    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3962933b-4378-4941-8983-d4ca1b616be1-istio-envoy\") pod \"e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl\" (UID: \"3962933b-4378-4941-8983-d4ca1b616be1\") " pod="openshift-ingress/e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl"
Sep 14 09:58:38.318839 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:38.318714    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3962933b-4378-4941-8983-d4ca1b616be1-credential-socket\") pod \"e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl\" (UID: \"3962933b-4378-4941-8983-d4ca1b616be1\") " pod="openshift-ingress/e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl"
Sep 14 09:58:38.318839 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:38.318739    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3962933b-4378-4941-8983-d4ca1b616be1-istio-token\") pod \"e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl\" (UID: \"3962933b-4378-4941-8983-d4ca1b616be1\") " pod="openshift-ingress/e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl"
Sep 14 09:58:38.419686 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:38.419641    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnhz8\" (UniqueName: \"kubernetes.io/projected/3962933b-4378-4941-8983-d4ca1b616be1-kube-api-access-lnhz8\") pod \"e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl\" (UID: \"3962933b-4378-4941-8983-d4ca1b616be1\") " pod="openshift-ingress/e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl"
Sep 14 09:58:38.419686 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:38.419684    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3962933b-4378-4941-8983-d4ca1b616be1-istio-envoy\") pod \"e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl\" (UID: \"3962933b-4378-4941-8983-d4ca1b616be1\") " pod="openshift-ingress/e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl"
Sep 14 09:58:38.419931 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:38.419791    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3962933b-4378-4941-8983-d4ca1b616be1-credential-socket\") pod \"e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl\" (UID: \"3962933b-4378-4941-8983-d4ca1b616be1\") " pod="openshift-ingress/e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl"
Sep 14 09:58:38.419931 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:38.419893    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3962933b-4378-4941-8983-d4ca1b616be1-istio-token\") pod \"e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl\" (UID: \"3962933b-4378-4941-8983-d4ca1b616be1\") " pod="openshift-ingress/e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl"
Sep 14 09:58:38.420067 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:38.420042    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3962933b-4378-4941-8983-d4ca1b616be1-istio-podinfo\") pod \"e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl\" (UID: \"3962933b-4378-4941-8983-d4ca1b616be1\") " pod="openshift-ingress/e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl"
Sep 14 09:58:38.420125 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:38.420070    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3962933b-4378-4941-8983-d4ca1b616be1-istio-data\") pod \"e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl\" (UID: \"3962933b-4378-4941-8983-d4ca1b616be1\") " pod="openshift-ingress/e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl"
Sep 14 09:58:38.420125 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:38.420101    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3962933b-4378-4941-8983-d4ca1b616be1-workload-socket\") pod \"e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl\" (UID: \"3962933b-4378-4941-8983-d4ca1b616be1\") " pod="openshift-ingress/e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl"
Sep 14 09:58:38.420241 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:38.420135    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3962933b-4378-4941-8983-d4ca1b616be1-workload-certs\") pod \"e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl\" (UID: \"3962933b-4378-4941-8983-d4ca1b616be1\") " pod="openshift-ingress/e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl"
Sep 14 09:58:38.420241 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:38.420204    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3962933b-4378-4941-8983-d4ca1b616be1-istiod-ca-cert\") pod \"e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl\" (UID: \"3962933b-4378-4941-8983-d4ca1b616be1\") " pod="openshift-ingress/e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl"
Sep 14 09:58:38.420454 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:38.420425    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3962933b-4378-4941-8983-d4ca1b616be1-credential-socket\") pod \"e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl\" (UID: \"3962933b-4378-4941-8983-d4ca1b616be1\") " pod="openshift-ingress/e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl"
Sep 14 09:58:38.420601 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:38.420561    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3962933b-4378-4941-8983-d4ca1b616be1-workload-socket\") pod \"e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl\" (UID: \"3962933b-4378-4941-8983-d4ca1b616be1\") " pod="openshift-ingress/e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl"
Sep 14 09:58:38.420666 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:38.420647    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3962933b-4378-4941-8983-d4ca1b616be1-istio-data\") pod \"e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl\" (UID: \"3962933b-4378-4941-8983-d4ca1b616be1\") " pod="openshift-ingress/e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl"
Sep 14 09:58:38.420731 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:38.420670    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3962933b-4378-4941-8983-d4ca1b616be1-workload-certs\") pod \"e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl\" (UID: \"3962933b-4378-4941-8983-d4ca1b616be1\") " pod="openshift-ingress/e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl"
Sep 14 09:58:38.420934 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:38.420912    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3962933b-4378-4941-8983-d4ca1b616be1-istiod-ca-cert\") pod \"e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl\" (UID: \"3962933b-4378-4941-8983-d4ca1b616be1\") " pod="openshift-ingress/e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl"
Sep 14 09:58:38.422395 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:38.422365    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3962933b-4378-4941-8983-d4ca1b616be1-istio-envoy\") pod \"e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl\" (UID: \"3962933b-4378-4941-8983-d4ca1b616be1\") " pod="openshift-ingress/e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl"
Sep 14 09:58:38.422616 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:38.422595    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3962933b-4378-4941-8983-d4ca1b616be1-istio-podinfo\") pod \"e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl\" (UID: \"3962933b-4378-4941-8983-d4ca1b616be1\") " pod="openshift-ingress/e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl"
Sep 14 09:58:38.430092 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:38.430046    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnhz8\" (UniqueName: \"kubernetes.io/projected/3962933b-4378-4941-8983-d4ca1b616be1-kube-api-access-lnhz8\") pod \"e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl\" (UID: \"3962933b-4378-4941-8983-d4ca1b616be1\") " pod="openshift-ingress/e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl"
Sep 14 09:58:38.431263 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:38.431233    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3962933b-4378-4941-8983-d4ca1b616be1-istio-token\") pod \"e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl\" (UID: \"3962933b-4378-4941-8983-d4ca1b616be1\") " pod="openshift-ingress/e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl"
Sep 14 09:58:38.520289 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:38.520256    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:58:38.531619 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:38.531581    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl"
Sep 14 09:58:38.814359 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:38.814327    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl"]
Sep 14 09:58:38.815142 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:38.815107    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:58:38.816183 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:58:38.816140    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3962933b_4378_4941_8983_d4ca1b616be1.slice/crio-09ddd8b90bd5c9b345a0732cc7aca0a28664149738a8c6cfc7252d7565d3b49c WatchSource:0}: Error finding container 09ddd8b90bd5c9b345a0732cc7aca0a28664149738a8c6cfc7252d7565d3b49c: Status 404 returned error can't find the container with id 09ddd8b90bd5c9b345a0732cc7aca0a28664149738a8c6cfc7252d7565d3b49c
Sep 14 09:58:38.818795 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:38.818761    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 09:58:38.818911 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:38.818849    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 09:58:38.818911 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:38.818895    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 09:58:39.499132 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:39.499093    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl" event={"ID":"3962933b-4378-4941-8983-d4ca1b616be1","Type":"ContainerStarted","Data":"ef3cd82a4ff7f6a430401e9722075656a95984b7949e0ff9b36a513f6fe7fd3d"}
Sep 14 09:58:39.499593 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:39.499153    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl" event={"ID":"3962933b-4378-4941-8983-d4ca1b616be1","Type":"ContainerStarted","Data":"09ddd8b90bd5c9b345a0732cc7aca0a28664149738a8c6cfc7252d7565d3b49c"}
Sep 14 09:58:39.528025 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:39.527932    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl" podStartSLOduration=1.5279118889999999 podStartE2EDuration="1.527911889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:58:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:58:39.524580724 +0000 UTC m=+908.564255346" watchObservedRunningTime="2026-09-14 09:58:39.527911889 +0000 UTC m=+908.567586507"
Sep 14 09:58:39.532037 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:39.531979    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl"
Sep 14 09:58:39.537890 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:39.537860    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl"
Sep 14 09:58:40.013388 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:40.013347    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:58:40.506105 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:40.506074    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl"
Sep 14 09:58:40.507210 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:40.507188    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl"
Sep 14 09:58:41.020593 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:41.020529    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:58:41.172299 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:41.172246    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:58:42.297969 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:42.297928    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:58:42.715402 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:42.715372    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq"
Sep 14 09:58:42.867089 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:42.866966    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cf814db-8f7c-437c-a8ca-f6bdd70330c5-istiod-ca-cert\") pod \"0cf814db-8f7c-437c-a8ca-f6bdd70330c5\" (UID: \"0cf814db-8f7c-437c-a8ca-f6bdd70330c5\") "
Sep 14 09:58:42.867089 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:42.867053    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cf814db-8f7c-437c-a8ca-f6bdd70330c5-istio-data\") pod \"0cf814db-8f7c-437c-a8ca-f6bdd70330c5\" (UID: \"0cf814db-8f7c-437c-a8ca-f6bdd70330c5\") "
Sep 14 09:58:42.867292 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:42.867131    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cf814db-8f7c-437c-a8ca-f6bdd70330c5-istio-podinfo\") pod \"0cf814db-8f7c-437c-a8ca-f6bdd70330c5\" (UID: \"0cf814db-8f7c-437c-a8ca-f6bdd70330c5\") "
Sep 14 09:58:42.867292 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:42.867172    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cf814db-8f7c-437c-a8ca-f6bdd70330c5-workload-certs\") pod \"0cf814db-8f7c-437c-a8ca-f6bdd70330c5\" (UID: \"0cf814db-8f7c-437c-a8ca-f6bdd70330c5\") "
Sep 14 09:58:42.867292 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:42.867204    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cf814db-8f7c-437c-a8ca-f6bdd70330c5-workload-socket\") pod \"0cf814db-8f7c-437c-a8ca-f6bdd70330c5\" (UID: \"0cf814db-8f7c-437c-a8ca-f6bdd70330c5\") "
Sep 14 09:58:42.867292 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:42.867266    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cf814db-8f7c-437c-a8ca-f6bdd70330c5-credential-socket\") pod \"0cf814db-8f7c-437c-a8ca-f6bdd70330c5\" (UID: \"0cf814db-8f7c-437c-a8ca-f6bdd70330c5\") "
Sep 14 09:58:42.867482 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:42.867315    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cf814db-8f7c-437c-a8ca-f6bdd70330c5-istio-token\") pod \"0cf814db-8f7c-437c-a8ca-f6bdd70330c5\" (UID: \"0cf814db-8f7c-437c-a8ca-f6bdd70330c5\") "
Sep 14 09:58:42.867482 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:42.867345    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tgddn\" (UniqueName: \"kubernetes.io/projected/0cf814db-8f7c-437c-a8ca-f6bdd70330c5-kube-api-access-tgddn\") pod \"0cf814db-8f7c-437c-a8ca-f6bdd70330c5\" (UID: \"0cf814db-8f7c-437c-a8ca-f6bdd70330c5\") "
Sep 14 09:58:42.867482 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:42.867373    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cf814db-8f7c-437c-a8ca-f6bdd70330c5-istio-envoy\") pod \"0cf814db-8f7c-437c-a8ca-f6bdd70330c5\" (UID: \"0cf814db-8f7c-437c-a8ca-f6bdd70330c5\") "
Sep 14 09:58:42.867482 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:42.867402    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cf814db-8f7c-437c-a8ca-f6bdd70330c5-istio-data" (OuterVolumeSpecName: "istio-data") pod "0cf814db-8f7c-437c-a8ca-f6bdd70330c5" (UID: "0cf814db-8f7c-437c-a8ca-f6bdd70330c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 09:58:42.867482 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:42.867421    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cf814db-8f7c-437c-a8ca-f6bdd70330c5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0cf814db-8f7c-437c-a8ca-f6bdd70330c5" (UID: "0cf814db-8f7c-437c-a8ca-f6bdd70330c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 09:58:42.867482 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:42.867436    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cf814db-8f7c-437c-a8ca-f6bdd70330c5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0cf814db-8f7c-437c-a8ca-f6bdd70330c5" (UID: "0cf814db-8f7c-437c-a8ca-f6bdd70330c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 09:58:42.867773 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:42.867570    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cf814db-8f7c-437c-a8ca-f6bdd70330c5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0cf814db-8f7c-437c-a8ca-f6bdd70330c5" (UID: "0cf814db-8f7c-437c-a8ca-f6bdd70330c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 09:58:42.867773 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:42.867598    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cf814db-8f7c-437c-a8ca-f6bdd70330c5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0cf814db-8f7c-437c-a8ca-f6bdd70330c5" (UID: "0cf814db-8f7c-437c-a8ca-f6bdd70330c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 09:58:42.867888 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:42.867800    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cf814db-8f7c-437c-a8ca-f6bdd70330c5-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:58:42.867888 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:42.867822    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cf814db-8f7c-437c-a8ca-f6bdd70330c5-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:58:42.867888 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:42.867837    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cf814db-8f7c-437c-a8ca-f6bdd70330c5-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:58:42.867888 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:42.867849    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cf814db-8f7c-437c-a8ca-f6bdd70330c5-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:58:42.867888 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:42.867863    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cf814db-8f7c-437c-a8ca-f6bdd70330c5-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:58:42.869556 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:42.869521    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cf814db-8f7c-437c-a8ca-f6bdd70330c5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0cf814db-8f7c-437c-a8ca-f6bdd70330c5" (UID: "0cf814db-8f7c-437c-a8ca-f6bdd70330c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 09:58:42.869556 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:42.869546    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cf814db-8f7c-437c-a8ca-f6bdd70330c5-istio-token" (OuterVolumeSpecName: "istio-token") pod "0cf814db-8f7c-437c-a8ca-f6bdd70330c5" (UID: "0cf814db-8f7c-437c-a8ca-f6bdd70330c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 09:58:42.869822 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:42.869794    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0cf814db-8f7c-437c-a8ca-f6bdd70330c5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0cf814db-8f7c-437c-a8ca-f6bdd70330c5" (UID: "0cf814db-8f7c-437c-a8ca-f6bdd70330c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 09:58:42.869822 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:42.869805    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cf814db-8f7c-437c-a8ca-f6bdd70330c5-kube-api-access-tgddn" (OuterVolumeSpecName: "kube-api-access-tgddn") pod "0cf814db-8f7c-437c-a8ca-f6bdd70330c5" (UID: "0cf814db-8f7c-437c-a8ca-f6bdd70330c5"). InnerVolumeSpecName "kube-api-access-tgddn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 09:58:42.965425 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:42.965382    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-2901ec/simulator-w4-2901ec-kserve-7c4cdb8dff-nhllq"]
Sep 14 09:58:42.965827 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:42.965809    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cf814db-8f7c-437c-a8ca-f6bdd70330c5" containerName="istio-proxy"
Sep 14 09:58:42.965875 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:42.965833    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cf814db-8f7c-437c-a8ca-f6bdd70330c5" containerName="istio-proxy"
Sep 14 09:58:42.965949 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:42.965936    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cf814db-8f7c-437c-a8ca-f6bdd70330c5" containerName="istio-proxy"
Sep 14 09:58:42.965990 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:42.965954    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cf814db-8f7c-437c-a8ca-f6bdd70330c5" containerName="istio-proxy"
Sep 14 09:58:42.968320 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:42.968288    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2901ec/simulator-w4-2901ec-kserve-7c4cdb8dff-nhllq"
Sep 14 09:58:42.968963 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:42.968934    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cf814db-8f7c-437c-a8ca-f6bdd70330c5-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:58:42.969112 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:42.968970    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgddn\" (UniqueName: \"kubernetes.io/projected/0cf814db-8f7c-437c-a8ca-f6bdd70330c5-kube-api-access-tgddn\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:58:42.969112 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:42.968986    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cf814db-8f7c-437c-a8ca-f6bdd70330c5-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:58:42.969112 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:42.969042    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cf814db-8f7c-437c-a8ca-f6bdd70330c5-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:58:42.970819 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:42.970789    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-2901ec\"/\"kube-root-ca.crt\""
Sep 14 09:58:42.970974 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:42.970957    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-2901ec\"/\"openshift-service-ca.crt\""
Sep 14 09:58:42.971086 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:42.971019    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-2901ec\"/\"default-dockercfg-2tscd\""
Sep 14 09:58:42.971185 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:42.971166    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-2901ec\"/\"simulator-w4-2901ec-kserve-self-signed-certs\""
Sep 14 09:58:42.983837 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:42.983797    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-2901ec/simulator-w4-2901ec-kserve-7c4cdb8dff-nhllq"]
Sep 14 09:58:43.017838 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:43.017800    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:58:43.070080 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:43.070042    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2q2x\" (UniqueName: \"kubernetes.io/projected/0d8c1e42-d4d8-456c-9cc1-1a493e8e3835-kube-api-access-c2q2x\") pod \"simulator-w4-2901ec-kserve-7c4cdb8dff-nhllq\" (UID: \"0d8c1e42-d4d8-456c-9cc1-1a493e8e3835\") " pod="e2e-models-e2e-worker-w4-2901ec/simulator-w4-2901ec-kserve-7c4cdb8dff-nhllq"
Sep 14 09:58:43.070269 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:43.070091    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d8c1e42-d4d8-456c-9cc1-1a493e8e3835-model-cache\") pod \"simulator-w4-2901ec-kserve-7c4cdb8dff-nhllq\" (UID: \"0d8c1e42-d4d8-456c-9cc1-1a493e8e3835\") " pod="e2e-models-e2e-worker-w4-2901ec/simulator-w4-2901ec-kserve-7c4cdb8dff-nhllq"
Sep 14 09:58:43.070269 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:43.070135    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d8c1e42-d4d8-456c-9cc1-1a493e8e3835-tls-certs\") pod \"simulator-w4-2901ec-kserve-7c4cdb8dff-nhllq\" (UID: \"0d8c1e42-d4d8-456c-9cc1-1a493e8e3835\") " pod="e2e-models-e2e-worker-w4-2901ec/simulator-w4-2901ec-kserve-7c4cdb8dff-nhllq"
Sep 14 09:58:43.070350 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:43.070269    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d8c1e42-d4d8-456c-9cc1-1a493e8e3835-dshm\") pod \"simulator-w4-2901ec-kserve-7c4cdb8dff-nhllq\" (UID: \"0d8c1e42-d4d8-456c-9cc1-1a493e8e3835\") " pod="e2e-models-e2e-worker-w4-2901ec/simulator-w4-2901ec-kserve-7c4cdb8dff-nhllq"
Sep 14 09:58:43.070350 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:43.070314    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d8c1e42-d4d8-456c-9cc1-1a493e8e3835-tmp-dir\") pod \"simulator-w4-2901ec-kserve-7c4cdb8dff-nhllq\" (UID: \"0d8c1e42-d4d8-456c-9cc1-1a493e8e3835\") " pod="e2e-models-e2e-worker-w4-2901ec/simulator-w4-2901ec-kserve-7c4cdb8dff-nhllq"
Sep 14 09:58:43.070440 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:43.070363    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d8c1e42-d4d8-456c-9cc1-1a493e8e3835-home\") pod \"simulator-w4-2901ec-kserve-7c4cdb8dff-nhllq\" (UID: \"0d8c1e42-d4d8-456c-9cc1-1a493e8e3835\") " pod="e2e-models-e2e-worker-w4-2901ec/simulator-w4-2901ec-kserve-7c4cdb8dff-nhllq"
Sep 14 09:58:43.171973 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:43.171862    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d8c1e42-d4d8-456c-9cc1-1a493e8e3835-dshm\") pod \"simulator-w4-2901ec-kserve-7c4cdb8dff-nhllq\" (UID: \"0d8c1e42-d4d8-456c-9cc1-1a493e8e3835\") " pod="e2e-models-e2e-worker-w4-2901ec/simulator-w4-2901ec-kserve-7c4cdb8dff-nhllq"
Sep 14 09:58:43.171973 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:43.171925    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d8c1e42-d4d8-456c-9cc1-1a493e8e3835-tmp-dir\") pod \"simulator-w4-2901ec-kserve-7c4cdb8dff-nhllq\" (UID: \"0d8c1e42-d4d8-456c-9cc1-1a493e8e3835\") " pod="e2e-models-e2e-worker-w4-2901ec/simulator-w4-2901ec-kserve-7c4cdb8dff-nhllq"
Sep 14 09:58:43.172255 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:43.171991    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d8c1e42-d4d8-456c-9cc1-1a493e8e3835-home\") pod \"simulator-w4-2901ec-kserve-7c4cdb8dff-nhllq\" (UID: \"0d8c1e42-d4d8-456c-9cc1-1a493e8e3835\") " pod="e2e-models-e2e-worker-w4-2901ec/simulator-w4-2901ec-kserve-7c4cdb8dff-nhllq"
Sep 14 09:58:43.172255 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:43.172136    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2q2x\" (UniqueName: \"kubernetes.io/projected/0d8c1e42-d4d8-456c-9cc1-1a493e8e3835-kube-api-access-c2q2x\") pod \"simulator-w4-2901ec-kserve-7c4cdb8dff-nhllq\" (UID: \"0d8c1e42-d4d8-456c-9cc1-1a493e8e3835\") " pod="e2e-models-e2e-worker-w4-2901ec/simulator-w4-2901ec-kserve-7c4cdb8dff-nhllq"
Sep 14 09:58:43.172255 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:43.172170    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d8c1e42-d4d8-456c-9cc1-1a493e8e3835-model-cache\") pod \"simulator-w4-2901ec-kserve-7c4cdb8dff-nhllq\" (UID: \"0d8c1e42-d4d8-456c-9cc1-1a493e8e3835\") " pod="e2e-models-e2e-worker-w4-2901ec/simulator-w4-2901ec-kserve-7c4cdb8dff-nhllq"
Sep 14 09:58:43.172255 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:43.172201    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d8c1e42-d4d8-456c-9cc1-1a493e8e3835-tls-certs\") pod \"simulator-w4-2901ec-kserve-7c4cdb8dff-nhllq\" (UID: \"0d8c1e42-d4d8-456c-9cc1-1a493e8e3835\") " pod="e2e-models-e2e-worker-w4-2901ec/simulator-w4-2901ec-kserve-7c4cdb8dff-nhllq"
Sep 14 09:58:43.172552 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:43.172423    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d8c1e42-d4d8-456c-9cc1-1a493e8e3835-tmp-dir\") pod \"simulator-w4-2901ec-kserve-7c4cdb8dff-nhllq\" (UID: \"0d8c1e42-d4d8-456c-9cc1-1a493e8e3835\") " pod="e2e-models-e2e-worker-w4-2901ec/simulator-w4-2901ec-kserve-7c4cdb8dff-nhllq"
Sep 14 09:58:43.172552 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:43.172509    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d8c1e42-d4d8-456c-9cc1-1a493e8e3835-home\") pod \"simulator-w4-2901ec-kserve-7c4cdb8dff-nhllq\" (UID: \"0d8c1e42-d4d8-456c-9cc1-1a493e8e3835\") " pod="e2e-models-e2e-worker-w4-2901ec/simulator-w4-2901ec-kserve-7c4cdb8dff-nhllq"
Sep 14 09:58:43.172658 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:43.172587    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d8c1e42-d4d8-456c-9cc1-1a493e8e3835-model-cache\") pod \"simulator-w4-2901ec-kserve-7c4cdb8dff-nhllq\" (UID: \"0d8c1e42-d4d8-456c-9cc1-1a493e8e3835\") " pod="e2e-models-e2e-worker-w4-2901ec/simulator-w4-2901ec-kserve-7c4cdb8dff-nhllq"
Sep 14 09:58:43.174557 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:43.174525    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d8c1e42-d4d8-456c-9cc1-1a493e8e3835-dshm\") pod \"simulator-w4-2901ec-kserve-7c4cdb8dff-nhllq\" (UID: \"0d8c1e42-d4d8-456c-9cc1-1a493e8e3835\") " pod="e2e-models-e2e-worker-w4-2901ec/simulator-w4-2901ec-kserve-7c4cdb8dff-nhllq"
Sep 14 09:58:43.174783 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:43.174765    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d8c1e42-d4d8-456c-9cc1-1a493e8e3835-tls-certs\") pod \"simulator-w4-2901ec-kserve-7c4cdb8dff-nhllq\" (UID: \"0d8c1e42-d4d8-456c-9cc1-1a493e8e3835\") " pod="e2e-models-e2e-worker-w4-2901ec/simulator-w4-2901ec-kserve-7c4cdb8dff-nhllq"
Sep 14 09:58:43.181243 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:43.181215    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2q2x\" (UniqueName: \"kubernetes.io/projected/0d8c1e42-d4d8-456c-9cc1-1a493e8e3835-kube-api-access-c2q2x\") pod \"simulator-w4-2901ec-kserve-7c4cdb8dff-nhllq\" (UID: \"0d8c1e42-d4d8-456c-9cc1-1a493e8e3835\") " pod="e2e-models-e2e-worker-w4-2901ec/simulator-w4-2901ec-kserve-7c4cdb8dff-nhllq"
Sep 14 09:58:43.282721 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:43.282674    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2901ec/simulator-w4-2901ec-kserve-7c4cdb8dff-nhllq"
Sep 14 09:58:43.446912 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:43.446749    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:58:43.525861 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:43.525822    2563 generic.go:356] "Generic (PLEG): container finished" podID="0cf814db-8f7c-437c-a8ca-f6bdd70330c5" containerID="9af890ecf7b5a47095b7c0331fc783227937554365348c3f1cd274b8cec3aa3b" exitCode=0
Sep 14 09:58:43.526062 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:43.525911    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq"
Sep 14 09:58:43.526062 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:43.525902    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq" event={"ID":"0cf814db-8f7c-437c-a8ca-f6bdd70330c5","Type":"ContainerDied","Data":"9af890ecf7b5a47095b7c0331fc783227937554365348c3f1cd274b8cec3aa3b"}
Sep 14 09:58:43.526062 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:43.526058    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq" event={"ID":"0cf814db-8f7c-437c-a8ca-f6bdd70330c5","Type":"ContainerDied","Data":"e83257e092e41ce5f759d07519bd6c65360f63975adc8ace0efb81190e2504b0"}
Sep 14 09:58:43.526191 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:43.526086    2563 scope.go:117] "RemoveContainer" containerID="9af890ecf7b5a47095b7c0331fc783227937554365348c3f1cd274b8cec3aa3b"
Sep 14 09:58:43.536013 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:43.535978    2563 scope.go:117] "RemoveContainer" containerID="2596ac840736ba3bd2896a6a839cca4b5032094e98d5660ae13a2c546048e143"
Sep 14 09:58:43.562689 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:43.560656    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq"]
Sep 14 09:58:43.565345 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:43.565308    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-2f4712d0-gw-openshift-default-dc484976-6kckq"]
Sep 14 09:58:43.570464 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:43.570425    2563 scope.go:117] "RemoveContainer" containerID="9af890ecf7b5a47095b7c0331fc783227937554365348c3f1cd274b8cec3aa3b"
Sep 14 09:58:43.570924 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:58:43.570870    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9af890ecf7b5a47095b7c0331fc783227937554365348c3f1cd274b8cec3aa3b\": container with ID starting with 9af890ecf7b5a47095b7c0331fc783227937554365348c3f1cd274b8cec3aa3b not found: ID does not exist" containerID="9af890ecf7b5a47095b7c0331fc783227937554365348c3f1cd274b8cec3aa3b"
Sep 14 09:58:43.570924 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:43.570915    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9af890ecf7b5a47095b7c0331fc783227937554365348c3f1cd274b8cec3aa3b"} err="failed to get container status \"9af890ecf7b5a47095b7c0331fc783227937554365348c3f1cd274b8cec3aa3b\": rpc error: code = NotFound desc = could not find container \"9af890ecf7b5a47095b7c0331fc783227937554365348c3f1cd274b8cec3aa3b\": container with ID starting with 9af890ecf7b5a47095b7c0331fc783227937554365348c3f1cd274b8cec3aa3b not found: ID does not exist"
Sep 14 09:58:43.571229 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:43.570935    2563 scope.go:117] "RemoveContainer" containerID="2596ac840736ba3bd2896a6a839cca4b5032094e98d5660ae13a2c546048e143"
Sep 14 09:58:43.571383 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:58:43.571364    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2596ac840736ba3bd2896a6a839cca4b5032094e98d5660ae13a2c546048e143\": container with ID starting with 2596ac840736ba3bd2896a6a839cca4b5032094e98d5660ae13a2c546048e143 not found: ID does not exist" containerID="2596ac840736ba3bd2896a6a839cca4b5032094e98d5660ae13a2c546048e143"
Sep 14 09:58:43.571383 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:43.571388    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2596ac840736ba3bd2896a6a839cca4b5032094e98d5660ae13a2c546048e143"} err="failed to get container status \"2596ac840736ba3bd2896a6a839cca4b5032094e98d5660ae13a2c546048e143\": rpc error: code = NotFound desc = could not find container \"2596ac840736ba3bd2896a6a839cca4b5032094e98d5660ae13a2c546048e143\": container with ID starting with 2596ac840736ba3bd2896a6a839cca4b5032094e98d5660ae13a2c546048e143 not found: ID does not exist"
Sep 14 09:58:43.571571 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:43.571403    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-2901ec/simulator-w4-2901ec-kserve-7c4cdb8dff-nhllq"]
Sep 14 09:58:43.573122 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:58:43.573095    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d8c1e42_d4d8_456c_9cc1_1a493e8e3835.slice/crio-1dd5b2106fc400efa45b6bbe87a10d08518f47e8a064c678ddb7bd5e7f743233 WatchSource:0}: Error finding container 1dd5b2106fc400efa45b6bbe87a10d08518f47e8a064c678ddb7bd5e7f743233: Status 404 returned error can't find the container with id 1dd5b2106fc400efa45b6bbe87a10d08518f47e8a064c678ddb7bd5e7f743233
Sep 14 09:58:44.294661 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:44.294617    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:58:44.395969 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:44.395933    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:58:44.535057 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:44.534980    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2901ec/simulator-w4-2901ec-kserve-7c4cdb8dff-nhllq" event={"ID":"0d8c1e42-d4d8-456c-9cc1-1a493e8e3835","Type":"ContainerStarted","Data":"3d2010bbad19cdef1c49b5bb819462c6d099207704c164d0ea3d76f8bd795e96"}
Sep 14 09:58:44.535466 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:44.535070    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2901ec/simulator-w4-2901ec-kserve-7c4cdb8dff-nhllq" event={"ID":"0d8c1e42-d4d8-456c-9cc1-1a493e8e3835","Type":"ContainerStarted","Data":"1dd5b2106fc400efa45b6bbe87a10d08518f47e8a064c678ddb7bd5e7f743233"}
Sep 14 09:58:44.554249 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:44.554113    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-2901ec/simulator-w4-2901ec-kserve-7c4cdb8dff-nhllq" podStartSLOduration=2.554092146 podStartE2EDuration="2.554092146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:58:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:58:44.551139501 +0000 UTC m=+913.590814122" watchObservedRunningTime="2026-09-14 09:58:44.554092146 +0000 UTC m=+913.593766834"
Sep 14 09:58:44.556306 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:44.556272    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:58:44.983370 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:44.983279    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:58:45.126494 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:45.126462    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:58:45.482846 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:45.482807    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0cf814db-8f7c-437c-a8ca-f6bdd70330c5" path="/var/lib/kubelet/pods/0cf814db-8f7c-437c-a8ca-f6bdd70330c5/volumes"
Sep 14 09:58:45.676552 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:45.676518    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:58:46.753304 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:46.753263    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:58:47.691963 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:47.691927    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:58:48.937655 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:48.937617    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:58:52.313521 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:52.313478    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:58:53.283135 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:53.283083    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2901ec/simulator-w4-2901ec-kserve-7c4cdb8dff-nhllq"
Sep 14 09:58:53.283472 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:53.283433    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-2901ec/simulator-w4-2901ec-kserve-7c4cdb8dff-nhllq"
Sep 14 09:58:53.296268 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:53.296234    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-2901ec/simulator-w4-2901ec-kserve-7c4cdb8dff-nhllq"
Sep 14 09:58:53.603040 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:58:53.602923    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2901ec/simulator-w4-2901ec-kserve-7c4cdb8dff-nhllq"
Sep 14 09:59:01.642331 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:01.642293    2563 generic.go:356] "Generic (PLEG): container finished" podID="3962933b-4378-4941-8983-d4ca1b616be1" containerID="ef3cd82a4ff7f6a430401e9722075656a95984b7949e0ff9b36a513f6fe7fd3d" exitCode=0
Sep 14 09:59:01.642806 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:01.642374    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl" event={"ID":"3962933b-4378-4941-8983-d4ca1b616be1","Type":"ContainerDied","Data":"ef3cd82a4ff7f6a430401e9722075656a95984b7949e0ff9b36a513f6fe7fd3d"}
Sep 14 09:59:01.642857 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:01.642810    2563 scope.go:117] "RemoveContainer" containerID="ef3cd82a4ff7f6a430401e9722075656a95984b7949e0ff9b36a513f6fe7fd3d"
Sep 14 09:59:01.644136 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:01.644106    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 09:59:01.644212 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:01.644189    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 09:59:01.644260 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:01.644223    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 09:59:02.651829 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:02.651794    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl" event={"ID":"3962933b-4378-4941-8983-d4ca1b616be1","Type":"ContainerStarted","Data":"2432695cc18bedcf009a74c87d0fd83d1dc509ffad962c4bcad1ebb3d11e3ee7"}
Sep 14 09:59:03.532574 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:03.532529    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl"
Sep 14 09:59:03.532922 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:03.532611    2563 patch_prober.go:28] interesting pod/e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 14 09:59:03.533147 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:03.533104    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl" podUID="3962933b-4378-4941-8983-d4ca1b616be1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 14 09:59:04.532478 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:04.532436    2563 patch_prober.go:28] interesting pod/e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 14 09:59:04.533134 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:04.533092    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl" podUID="3962933b-4378-4941-8983-d4ca1b616be1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 14 09:59:05.532813 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:05.532768    2563 patch_prober.go:28] interesting pod/e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 14 09:59:05.533497 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:05.533454    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl" podUID="3962933b-4378-4941-8983-d4ca1b616be1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 14 09:59:05.727618 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:05.727571    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:59:06.533220 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:06.532757    2563 patch_prober.go:28] interesting pod/e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 14 09:59:06.533220 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:06.532821    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl" podUID="3962933b-4378-4941-8983-d4ca1b616be1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 14 09:59:07.160717 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:07.160679    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:59:07.532987 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:07.532941    2563 patch_prober.go:28] interesting pod/e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 14 09:59:07.533339 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:07.533307    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl" podUID="3962933b-4378-4941-8983-d4ca1b616be1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 14 09:59:08.057588 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:08.057555    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:59:08.532979 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:08.532928    2563 patch_prober.go:28] interesting pod/e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 14 09:59:08.533255 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:08.532987    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl" podUID="3962933b-4378-4941-8983-d4ca1b616be1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 14 09:59:08.890739 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:08.890644    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:59:09.536535 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:09.536493    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl"
Sep 14 09:59:09.536805 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:09.536783    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl"
Sep 14 09:59:09.537602 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:09.537581    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl"
Sep 14 09:59:09.667318 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:09.667282    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:59:10.412523 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:10.412484    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:59:11.252751 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:11.252713    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:59:11.762576 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:11.762473    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7fe0e18-7d76h"]
Sep 14 09:59:11.762988 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:11.762925    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cf814db-8f7c-437c-a8ca-f6bdd70330c5" containerName="istio-proxy"
Sep 14 09:59:11.762988 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:11.762940    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cf814db-8f7c-437c-a8ca-f6bdd70330c5" containerName="istio-proxy"
Sep 14 09:59:11.765401 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:11.765380    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7fe0e18-7d76h"
Sep 14 09:59:11.768030 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:11.767970    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 14 09:59:11.768182 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:11.768053    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gf4xj\""
Sep 14 09:59:11.775979 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:11.775943    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7fe0e18-7d76h"]
Sep 14 09:59:11.870787 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:11.870747    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c57cc384-15aa-4425-9ffc-27461fa9aa30-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e7fe0e18-7d76h\" (UID: \"c57cc384-15aa-4425-9ffc-27461fa9aa30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7fe0e18-7d76h"
Sep 14 09:59:11.972213 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:11.972171    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c57cc384-15aa-4425-9ffc-27461fa9aa30-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e7fe0e18-7d76h\" (UID: \"c57cc384-15aa-4425-9ffc-27461fa9aa30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7fe0e18-7d76h"
Sep 14 09:59:11.972861 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:11.972832    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c57cc384-15aa-4425-9ffc-27461fa9aa30-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e7fe0e18-7d76h\" (UID: \"c57cc384-15aa-4425-9ffc-27461fa9aa30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7fe0e18-7d76h"
Sep 14 09:59:12.078635 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:12.078538    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7fe0e18-7d76h"
Sep 14 09:59:12.353690 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:12.353652    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7fe0e18-7d76h"]
Sep 14 09:59:12.355378 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:59:12.355338    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc57cc384_15aa_4425_9ffc_27461fa9aa30.slice/crio-852e194d45af341c9fb0bef0de01006cf45333ffaae89c89321deefdf0315a4e WatchSource:0}: Error finding container 852e194d45af341c9fb0bef0de01006cf45333ffaae89c89321deefdf0315a4e: Status 404 returned error can't find the container with id 852e194d45af341c9fb0bef0de01006cf45333ffaae89c89321deefdf0315a4e
Sep 14 09:59:12.494927 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:12.494881    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:59:12.712087 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:12.711955    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7fe0e18-7d76h" event={"ID":"c57cc384-15aa-4425-9ffc-27461fa9aa30","Type":"ContainerStarted","Data":"98e61729d23719c4c2922bac670fe143f17e5be70b98a591ae4ac2d04e8c77b5"}
Sep 14 09:59:12.712087 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:12.712047    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7fe0e18-7d76h" event={"ID":"c57cc384-15aa-4425-9ffc-27461fa9aa30","Type":"ContainerStarted","Data":"852e194d45af341c9fb0bef0de01006cf45333ffaae89c89321deefdf0315a4e"}
Sep 14 09:59:12.732803 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:12.732743    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7fe0e18-7d76h" podStartSLOduration=1.732727822 podStartE2EDuration="1.732727822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:59:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:59:12.7323063 +0000 UTC m=+941.771980922" watchObservedRunningTime="2026-09-14 09:59:12.732727822 +0000 UTC m=+941.772402443"
Sep 14 09:59:13.134303 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:13.134251    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-b6812e/simulator-w0-b6812e-kserve-6c5dd8795-rvbtb"]
Sep 14 09:59:13.137035 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:13.136990    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b6812e/simulator-w0-b6812e-kserve-6c5dd8795-rvbtb"
Sep 14 09:59:13.140210 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:13.140181    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b6812e\"/\"default-dockercfg-grxrg\""
Sep 14 09:59:13.140592 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:13.140571    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-b6812e\"/\"kube-root-ca.crt\""
Sep 14 09:59:13.140734 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:13.140611    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-b6812e\"/\"openshift-service-ca.crt\""
Sep 14 09:59:13.140875 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:13.140858    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b6812e\"/\"simulator-w0-b6812e-kserve-self-signed-certs\""
Sep 14 09:59:13.152570 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:13.152533    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b6812e/simulator-w0-b6812e-kserve-6c5dd8795-rvbtb"]
Sep 14 09:59:13.191540 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:13.191502    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:59:13.287204 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:13.287101    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6fd0d32-d8ff-411e-8183-502161d51def-dshm\") pod \"simulator-w0-b6812e-kserve-6c5dd8795-rvbtb\" (UID: \"c6fd0d32-d8ff-411e-8183-502161d51def\") " pod="e2e-models-e2e-worker-w0-b6812e/simulator-w0-b6812e-kserve-6c5dd8795-rvbtb"
Sep 14 09:59:13.287204 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:13.287154    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6fd0d32-d8ff-411e-8183-502161d51def-tmp-dir\") pod \"simulator-w0-b6812e-kserve-6c5dd8795-rvbtb\" (UID: \"c6fd0d32-d8ff-411e-8183-502161d51def\") " pod="e2e-models-e2e-worker-w0-b6812e/simulator-w0-b6812e-kserve-6c5dd8795-rvbtb"
Sep 14 09:59:13.287450 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:13.287218    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6fd0d32-d8ff-411e-8183-502161d51def-model-cache\") pod \"simulator-w0-b6812e-kserve-6c5dd8795-rvbtb\" (UID: \"c6fd0d32-d8ff-411e-8183-502161d51def\") " pod="e2e-models-e2e-worker-w0-b6812e/simulator-w0-b6812e-kserve-6c5dd8795-rvbtb"
Sep 14 09:59:13.287450 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:13.287241    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ndhs\" (UniqueName: \"kubernetes.io/projected/c6fd0d32-d8ff-411e-8183-502161d51def-kube-api-access-2ndhs\") pod \"simulator-w0-b6812e-kserve-6c5dd8795-rvbtb\" (UID: \"c6fd0d32-d8ff-411e-8183-502161d51def\") " pod="e2e-models-e2e-worker-w0-b6812e/simulator-w0-b6812e-kserve-6c5dd8795-rvbtb"
Sep 14 09:59:13.287450 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:13.287360    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6fd0d32-d8ff-411e-8183-502161d51def-tls-certs\") pod \"simulator-w0-b6812e-kserve-6c5dd8795-rvbtb\" (UID: \"c6fd0d32-d8ff-411e-8183-502161d51def\") " pod="e2e-models-e2e-worker-w0-b6812e/simulator-w0-b6812e-kserve-6c5dd8795-rvbtb"
Sep 14 09:59:13.287591 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:13.287467    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6fd0d32-d8ff-411e-8183-502161d51def-home\") pod \"simulator-w0-b6812e-kserve-6c5dd8795-rvbtb\" (UID: \"c6fd0d32-d8ff-411e-8183-502161d51def\") " pod="e2e-models-e2e-worker-w0-b6812e/simulator-w0-b6812e-kserve-6c5dd8795-rvbtb"
Sep 14 09:59:13.388665 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:13.388626    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6fd0d32-d8ff-411e-8183-502161d51def-tmp-dir\") pod \"simulator-w0-b6812e-kserve-6c5dd8795-rvbtb\" (UID: \"c6fd0d32-d8ff-411e-8183-502161d51def\") " pod="e2e-models-e2e-worker-w0-b6812e/simulator-w0-b6812e-kserve-6c5dd8795-rvbtb"
Sep 14 09:59:13.388891 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:13.388688    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6fd0d32-d8ff-411e-8183-502161d51def-model-cache\") pod \"simulator-w0-b6812e-kserve-6c5dd8795-rvbtb\" (UID: \"c6fd0d32-d8ff-411e-8183-502161d51def\") " pod="e2e-models-e2e-worker-w0-b6812e/simulator-w0-b6812e-kserve-6c5dd8795-rvbtb"
Sep 14 09:59:13.388891 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:13.388712    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ndhs\" (UniqueName: \"kubernetes.io/projected/c6fd0d32-d8ff-411e-8183-502161d51def-kube-api-access-2ndhs\") pod \"simulator-w0-b6812e-kserve-6c5dd8795-rvbtb\" (UID: \"c6fd0d32-d8ff-411e-8183-502161d51def\") " pod="e2e-models-e2e-worker-w0-b6812e/simulator-w0-b6812e-kserve-6c5dd8795-rvbtb"
Sep 14 09:59:13.388891 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:13.388764    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6fd0d32-d8ff-411e-8183-502161d51def-tls-certs\") pod \"simulator-w0-b6812e-kserve-6c5dd8795-rvbtb\" (UID: \"c6fd0d32-d8ff-411e-8183-502161d51def\") " pod="e2e-models-e2e-worker-w0-b6812e/simulator-w0-b6812e-kserve-6c5dd8795-rvbtb"
Sep 14 09:59:13.388891 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:13.388814    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6fd0d32-d8ff-411e-8183-502161d51def-home\") pod \"simulator-w0-b6812e-kserve-6c5dd8795-rvbtb\" (UID: \"c6fd0d32-d8ff-411e-8183-502161d51def\") " pod="e2e-models-e2e-worker-w0-b6812e/simulator-w0-b6812e-kserve-6c5dd8795-rvbtb"
Sep 14 09:59:13.388891 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:13.388857    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6fd0d32-d8ff-411e-8183-502161d51def-dshm\") pod \"simulator-w0-b6812e-kserve-6c5dd8795-rvbtb\" (UID: \"c6fd0d32-d8ff-411e-8183-502161d51def\") " pod="e2e-models-e2e-worker-w0-b6812e/simulator-w0-b6812e-kserve-6c5dd8795-rvbtb"
Sep 14 09:59:13.389194 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:13.389162    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6fd0d32-d8ff-411e-8183-502161d51def-tmp-dir\") pod \"simulator-w0-b6812e-kserve-6c5dd8795-rvbtb\" (UID: \"c6fd0d32-d8ff-411e-8183-502161d51def\") " pod="e2e-models-e2e-worker-w0-b6812e/simulator-w0-b6812e-kserve-6c5dd8795-rvbtb"
Sep 14 09:59:13.389248 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:13.389196    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6fd0d32-d8ff-411e-8183-502161d51def-model-cache\") pod \"simulator-w0-b6812e-kserve-6c5dd8795-rvbtb\" (UID: \"c6fd0d32-d8ff-411e-8183-502161d51def\") " pod="e2e-models-e2e-worker-w0-b6812e/simulator-w0-b6812e-kserve-6c5dd8795-rvbtb"
Sep 14 09:59:13.389287 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:13.389241    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6fd0d32-d8ff-411e-8183-502161d51def-home\") pod \"simulator-w0-b6812e-kserve-6c5dd8795-rvbtb\" (UID: \"c6fd0d32-d8ff-411e-8183-502161d51def\") " pod="e2e-models-e2e-worker-w0-b6812e/simulator-w0-b6812e-kserve-6c5dd8795-rvbtb"
Sep 14 09:59:13.391333 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:13.391311    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6fd0d32-d8ff-411e-8183-502161d51def-dshm\") pod \"simulator-w0-b6812e-kserve-6c5dd8795-rvbtb\" (UID: \"c6fd0d32-d8ff-411e-8183-502161d51def\") " pod="e2e-models-e2e-worker-w0-b6812e/simulator-w0-b6812e-kserve-6c5dd8795-rvbtb"
Sep 14 09:59:13.391523 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:13.391506    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6fd0d32-d8ff-411e-8183-502161d51def-tls-certs\") pod \"simulator-w0-b6812e-kserve-6c5dd8795-rvbtb\" (UID: \"c6fd0d32-d8ff-411e-8183-502161d51def\") " pod="e2e-models-e2e-worker-w0-b6812e/simulator-w0-b6812e-kserve-6c5dd8795-rvbtb"
Sep 14 09:59:13.397735 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:13.397706    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ndhs\" (UniqueName: \"kubernetes.io/projected/c6fd0d32-d8ff-411e-8183-502161d51def-kube-api-access-2ndhs\") pod \"simulator-w0-b6812e-kserve-6c5dd8795-rvbtb\" (UID: \"c6fd0d32-d8ff-411e-8183-502161d51def\") " pod="e2e-models-e2e-worker-w0-b6812e/simulator-w0-b6812e-kserve-6c5dd8795-rvbtb"
Sep 14 09:59:13.447694 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:13.447654    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b6812e/simulator-w0-b6812e-kserve-6c5dd8795-rvbtb"
Sep 14 09:59:13.605569 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:13.605471    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:59:13.720468 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:13.720428    2563 generic.go:356] "Generic (PLEG): container finished" podID="c57cc384-15aa-4425-9ffc-27461fa9aa30" containerID="98e61729d23719c4c2922bac670fe143f17e5be70b98a591ae4ac2d04e8c77b5" exitCode=0
Sep 14 09:59:13.720654 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:13.720508    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7fe0e18-7d76h" event={"ID":"c57cc384-15aa-4425-9ffc-27461fa9aa30","Type":"ContainerDied","Data":"98e61729d23719c4c2922bac670fe143f17e5be70b98a591ae4ac2d04e8c77b5"}
Sep 14 09:59:13.738012 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:13.737976    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b6812e/simulator-w0-b6812e-kserve-6c5dd8795-rvbtb"]
Sep 14 09:59:13.741736 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:59:13.741686    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc6fd0d32_d8ff_411e_8183_502161d51def.slice/crio-42c781a639f159fadc48bb4239cda3fa0696f41e54bf786216e4a1e76db839a7 WatchSource:0}: Error finding container 42c781a639f159fadc48bb4239cda3fa0696f41e54bf786216e4a1e76db839a7: Status 404 returned error can't find the container with id 42c781a639f159fadc48bb4239cda3fa0696f41e54bf786216e4a1e76db839a7
Sep 14 09:59:14.647185 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:14.647144    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:59:14.728411 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:14.728374    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b6812e/simulator-w0-b6812e-kserve-6c5dd8795-rvbtb" event={"ID":"c6fd0d32-d8ff-411e-8183-502161d51def","Type":"ContainerStarted","Data":"6b6fd648822e3e5ffdd9c6baae0d4e6b23f97f2c0b7f7ffd28bd5f3148c52e69"}
Sep 14 09:59:14.728411 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:14.728420    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b6812e/simulator-w0-b6812e-kserve-6c5dd8795-rvbtb" event={"ID":"c6fd0d32-d8ff-411e-8183-502161d51def","Type":"ContainerStarted","Data":"42c781a639f159fadc48bb4239cda3fa0696f41e54bf786216e4a1e76db839a7"}
Sep 14 09:59:14.750590 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:14.750529    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-b6812e/simulator-w0-b6812e-kserve-6c5dd8795-rvbtb" podStartSLOduration=1.750509953 podStartE2EDuration="1.750509953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:59:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:59:14.74883852 +0000 UTC m=+943.788513141" watchObservedRunningTime="2026-09-14 09:59:14.750509953 +0000 UTC m=+943.790184574"
Sep 14 09:59:14.875602 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:14.875568    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7fe0e18-7d76h"
Sep 14 09:59:15.004369 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:15.004325    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c57cc384-15aa-4425-9ffc-27461fa9aa30-maas-api-service-ca\") pod \"c57cc384-15aa-4425-9ffc-27461fa9aa30\" (UID: \"c57cc384-15aa-4425-9ffc-27461fa9aa30\") "
Sep 14 09:59:15.004756 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:15.004729    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c57cc384-15aa-4425-9ffc-27461fa9aa30-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c57cc384-15aa-4425-9ffc-27461fa9aa30" (UID: "c57cc384-15aa-4425-9ffc-27461fa9aa30"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 09:59:15.105826 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:15.105776    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c57cc384-15aa-4425-9ffc-27461fa9aa30-maas-api-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:59:15.604486 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:15.604452    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:59:15.735743 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:15.735709    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7fe0e18-7d76h"
Sep 14 09:59:15.736293 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:15.735748    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7fe0e18-7d76h" event={"ID":"c57cc384-15aa-4425-9ffc-27461fa9aa30","Type":"ContainerDied","Data":"852e194d45af341c9fb0bef0de01006cf45333ffaae89c89321deefdf0315a4e"}
Sep 14 09:59:15.736293 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:15.735796    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="852e194d45af341c9fb0bef0de01006cf45333ffaae89c89321deefdf0315a4e"
Sep 14 09:59:16.428259 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:16.428221    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:59:17.227143 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:17.227105    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:59:17.974156 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:17.974114    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:59:22.607678 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:22.607635    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:59:23.326052 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:23.326012    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:59:23.443697 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:23.443652    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:59:23.448581 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:23.448554    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-b6812e/simulator-w0-b6812e-kserve-6c5dd8795-rvbtb"
Sep 14 09:59:23.448739 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:23.448607    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b6812e/simulator-w0-b6812e-kserve-6c5dd8795-rvbtb"
Sep 14 09:59:23.461446 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:23.461411    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-b6812e/simulator-w0-b6812e-kserve-6c5dd8795-rvbtb"
Sep 14 09:59:23.651831 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:23.651740    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7fe0e18-7d76h"]
Sep 14 09:59:23.652623 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:23.652596    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:59:23.656420 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:23.656369    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7fe0e18-7d76h"]
Sep 14 09:59:23.734936 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:23.734906    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:59:23.797933 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:23.797900    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b6812e/simulator-w0-b6812e-kserve-6c5dd8795-rvbtb"
Sep 14 09:59:24.115929 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:24.115896    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:59:24.763864 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:24.763823    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl"]
Sep 14 09:59:24.764358 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:24.764077    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl" podUID="3962933b-4378-4941-8983-d4ca1b616be1" containerName="istio-proxy" containerID="cri-o://2432695cc18bedcf009a74c87d0fd83d1dc509ffad962c4bcad1ebb3d11e3ee7" gracePeriod=30
Sep 14 09:59:24.772574 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:24.772522    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:59:25.124117 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:25.124030    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:59:25.235175 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:25.235138    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:59:25.482502 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:25.482410    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c57cc384-15aa-4425-9ffc-27461fa9aa30" path="/var/lib/kubelet/pods/c57cc384-15aa-4425-9ffc-27461fa9aa30/volumes"
Sep 14 09:59:25.556871 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:25.556820    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm"]
Sep 14 09:59:25.557348 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:25.557329    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c57cc384-15aa-4425-9ffc-27461fa9aa30" containerName="revoke-keys"
Sep 14 09:59:25.557348 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:25.557349    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c57cc384-15aa-4425-9ffc-27461fa9aa30" containerName="revoke-keys"
Sep 14 09:59:25.557525 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:25.557445    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c57cc384-15aa-4425-9ffc-27461fa9aa30" containerName="revoke-keys"
Sep 14 09:59:25.559907 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:25.559880    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm"
Sep 14 09:59:25.565788 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:25.565753    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-80fa7537-openshift-default-dockercfg-hccb4\""
Sep 14 09:59:25.576969 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:25.576930    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm"]
Sep 14 09:59:25.590372 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:25.590341    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:59:25.717494 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:25.717459    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/309ac5a4-3502-4351-830d-e6cf50ab73be-istiod-ca-cert\") pod \"e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm\" (UID: \"309ac5a4-3502-4351-830d-e6cf50ab73be\") " pod="openshift-ingress/e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm"
Sep 14 09:59:25.717494 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:25.717499    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/309ac5a4-3502-4351-830d-e6cf50ab73be-istio-envoy\") pod \"e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm\" (UID: \"309ac5a4-3502-4351-830d-e6cf50ab73be\") " pod="openshift-ingress/e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm"
Sep 14 09:59:25.717780 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:25.717585    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67tqf\" (UniqueName: \"kubernetes.io/projected/309ac5a4-3502-4351-830d-e6cf50ab73be-kube-api-access-67tqf\") pod \"e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm\" (UID: \"309ac5a4-3502-4351-830d-e6cf50ab73be\") " pod="openshift-ingress/e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm"
Sep 14 09:59:25.717780 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:25.717697    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/309ac5a4-3502-4351-830d-e6cf50ab73be-istio-podinfo\") pod \"e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm\" (UID: \"309ac5a4-3502-4351-830d-e6cf50ab73be\") " pod="openshift-ingress/e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm"
Sep 14 09:59:25.717780 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:25.717730    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/309ac5a4-3502-4351-830d-e6cf50ab73be-istio-data\") pod \"e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm\" (UID: \"309ac5a4-3502-4351-830d-e6cf50ab73be\") " pod="openshift-ingress/e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm"
Sep 14 09:59:25.717780 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:25.717755    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/309ac5a4-3502-4351-830d-e6cf50ab73be-workload-certs\") pod \"e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm\" (UID: \"309ac5a4-3502-4351-830d-e6cf50ab73be\") " pod="openshift-ingress/e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm"
Sep 14 09:59:25.717920 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:25.717798    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/309ac5a4-3502-4351-830d-e6cf50ab73be-credential-socket\") pod \"e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm\" (UID: \"309ac5a4-3502-4351-830d-e6cf50ab73be\") " pod="openshift-ingress/e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm"
Sep 14 09:59:25.717920 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:25.717827    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/309ac5a4-3502-4351-830d-e6cf50ab73be-istio-token\") pod \"e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm\" (UID: \"309ac5a4-3502-4351-830d-e6cf50ab73be\") " pod="openshift-ingress/e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm"
Sep 14 09:59:25.717920 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:25.717869    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/309ac5a4-3502-4351-830d-e6cf50ab73be-workload-socket\") pod \"e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm\" (UID: \"309ac5a4-3502-4351-830d-e6cf50ab73be\") " pod="openshift-ingress/e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm"
Sep 14 09:59:25.797700 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:25.797663    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:59:25.818916 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:25.818867    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/309ac5a4-3502-4351-830d-e6cf50ab73be-istiod-ca-cert\") pod \"e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm\" (UID: \"309ac5a4-3502-4351-830d-e6cf50ab73be\") " pod="openshift-ingress/e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm"
Sep 14 09:59:25.819183 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:25.818927    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/309ac5a4-3502-4351-830d-e6cf50ab73be-istio-envoy\") pod \"e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm\" (UID: \"309ac5a4-3502-4351-830d-e6cf50ab73be\") " pod="openshift-ingress/e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm"
Sep 14 09:59:25.819183 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:25.819059    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67tqf\" (UniqueName: \"kubernetes.io/projected/309ac5a4-3502-4351-830d-e6cf50ab73be-kube-api-access-67tqf\") pod \"e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm\" (UID: \"309ac5a4-3502-4351-830d-e6cf50ab73be\") " pod="openshift-ingress/e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm"
Sep 14 09:59:25.819183 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:25.819121    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/309ac5a4-3502-4351-830d-e6cf50ab73be-istio-podinfo\") pod \"e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm\" (UID: \"309ac5a4-3502-4351-830d-e6cf50ab73be\") " pod="openshift-ingress/e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm"
Sep 14 09:59:25.819183 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:25.819150    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/309ac5a4-3502-4351-830d-e6cf50ab73be-istio-data\") pod \"e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm\" (UID: \"309ac5a4-3502-4351-830d-e6cf50ab73be\") " pod="openshift-ingress/e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm"
Sep 14 09:59:25.819412 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:25.819185    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/309ac5a4-3502-4351-830d-e6cf50ab73be-workload-certs\") pod \"e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm\" (UID: \"309ac5a4-3502-4351-830d-e6cf50ab73be\") " pod="openshift-ingress/e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm"
Sep 14 09:59:25.819412 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:25.819227    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/309ac5a4-3502-4351-830d-e6cf50ab73be-credential-socket\") pod \"e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm\" (UID: \"309ac5a4-3502-4351-830d-e6cf50ab73be\") " pod="openshift-ingress/e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm"
Sep 14 09:59:25.819412 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:25.819276    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/309ac5a4-3502-4351-830d-e6cf50ab73be-istio-token\") pod \"e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm\" (UID: \"309ac5a4-3502-4351-830d-e6cf50ab73be\") " pod="openshift-ingress/e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm"
Sep 14 09:59:25.819412 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:25.819324    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/309ac5a4-3502-4351-830d-e6cf50ab73be-workload-socket\") pod \"e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm\" (UID: \"309ac5a4-3502-4351-830d-e6cf50ab73be\") " pod="openshift-ingress/e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm"
Sep 14 09:59:25.819619 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:25.819498    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/309ac5a4-3502-4351-830d-e6cf50ab73be-istio-data\") pod \"e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm\" (UID: \"309ac5a4-3502-4351-830d-e6cf50ab73be\") " pod="openshift-ingress/e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm"
Sep 14 09:59:25.819707 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:25.819677    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/309ac5a4-3502-4351-830d-e6cf50ab73be-credential-socket\") pod \"e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm\" (UID: \"309ac5a4-3502-4351-830d-e6cf50ab73be\") " pod="openshift-ingress/e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm"
Sep 14 09:59:25.819773 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:25.819734    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/309ac5a4-3502-4351-830d-e6cf50ab73be-workload-socket\") pod \"e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm\" (UID: \"309ac5a4-3502-4351-830d-e6cf50ab73be\") " pod="openshift-ingress/e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm"
Sep 14 09:59:25.819773 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:25.819739    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/309ac5a4-3502-4351-830d-e6cf50ab73be-istiod-ca-cert\") pod \"e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm\" (UID: \"309ac5a4-3502-4351-830d-e6cf50ab73be\") " pod="openshift-ingress/e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm"
Sep 14 09:59:25.819773 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:25.819753    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/309ac5a4-3502-4351-830d-e6cf50ab73be-workload-certs\") pod \"e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm\" (UID: \"309ac5a4-3502-4351-830d-e6cf50ab73be\") " pod="openshift-ingress/e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm"
Sep 14 09:59:25.821915 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:25.821895    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/309ac5a4-3502-4351-830d-e6cf50ab73be-istio-envoy\") pod \"e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm\" (UID: \"309ac5a4-3502-4351-830d-e6cf50ab73be\") " pod="openshift-ingress/e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm"
Sep 14 09:59:25.822104 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:25.822085    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/309ac5a4-3502-4351-830d-e6cf50ab73be-istio-podinfo\") pod \"e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm\" (UID: \"309ac5a4-3502-4351-830d-e6cf50ab73be\") " pod="openshift-ingress/e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm"
Sep 14 09:59:25.827860 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:25.827821    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67tqf\" (UniqueName: \"kubernetes.io/projected/309ac5a4-3502-4351-830d-e6cf50ab73be-kube-api-access-67tqf\") pod \"e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm\" (UID: \"309ac5a4-3502-4351-830d-e6cf50ab73be\") " pod="openshift-ingress/e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm"
Sep 14 09:59:25.827860 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:25.827836    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/309ac5a4-3502-4351-830d-e6cf50ab73be-istio-token\") pod \"e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm\" (UID: \"309ac5a4-3502-4351-830d-e6cf50ab73be\") " pod="openshift-ingress/e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm"
Sep 14 09:59:25.871613 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:25.871566    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm"
Sep 14 09:59:26.042174 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:26.042131    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:59:26.155214 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:26.155184    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm"]
Sep 14 09:59:26.156223 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:59:26.156190    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod309ac5a4_3502_4351_830d_e6cf50ab73be.slice/crio-7a7f09c6bc969f5eac3b1d8f51b171cd8233c9de2423ab042dfe33c9a1674ae1 WatchSource:0}: Error finding container 7a7f09c6bc969f5eac3b1d8f51b171cd8233c9de2423ab042dfe33c9a1674ae1: Status 404 returned error can't find the container with id 7a7f09c6bc969f5eac3b1d8f51b171cd8233c9de2423ab042dfe33c9a1674ae1
Sep 14 09:59:26.159035 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:26.158981    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 09:59:26.159120 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:26.159091    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 09:59:26.159158 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:26.159130    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 09:59:26.806637 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:26.806588    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm" event={"ID":"309ac5a4-3502-4351-830d-e6cf50ab73be","Type":"ContainerStarted","Data":"f9fabeae6867291a9e9ca5da8d5aef251af47ecb90ac1fe8f7c7de5619ff90a3"}
Sep 14 09:59:26.807061 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:26.806698    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm" event={"ID":"309ac5a4-3502-4351-830d-e6cf50ab73be","Type":"ContainerStarted","Data":"7a7f09c6bc969f5eac3b1d8f51b171cd8233c9de2423ab042dfe33c9a1674ae1"}
Sep 14 09:59:26.836944 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:26.836880    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm" podStartSLOduration=1.836861872 podStartE2EDuration="1.836861872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 09:59:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 09:59:26.836435575 +0000 UTC m=+955.876110198" watchObservedRunningTime="2026-09-14 09:59:26.836861872 +0000 UTC m=+955.876536493"
Sep 14 09:59:26.872649 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:26.872610    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm"
Sep 14 09:59:26.998785 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:26.998749    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:59:27.144090 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:27.143963    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:59:27.358215 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:27.358184    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:59:27.689336 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:27.689298    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:59:27.878096 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:27.878063    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm"
Sep 14 09:59:28.307353 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:28.307319    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:59:28.447055 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:28.446980    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:59:28.596665 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:28.596570    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:59:28.822483 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:28.822448    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm"
Sep 14 09:59:28.823633 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:28.823612    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm"
Sep 14 09:59:29.682361 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:29.682323    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:59:30.032639 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:30.032607    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl"
Sep 14 09:59:30.067079 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:30.067044    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3962933b-4378-4941-8983-d4ca1b616be1-credential-socket\") pod \"3962933b-4378-4941-8983-d4ca1b616be1\" (UID: \"3962933b-4378-4941-8983-d4ca1b616be1\") "
Sep 14 09:59:30.067079 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:30.067088    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3962933b-4378-4941-8983-d4ca1b616be1-istiod-ca-cert\") pod \"3962933b-4378-4941-8983-d4ca1b616be1\" (UID: \"3962933b-4378-4941-8983-d4ca1b616be1\") "
Sep 14 09:59:30.067347 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:30.067128    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3962933b-4378-4941-8983-d4ca1b616be1-istio-token\") pod \"3962933b-4378-4941-8983-d4ca1b616be1\" (UID: \"3962933b-4378-4941-8983-d4ca1b616be1\") "
Sep 14 09:59:30.067347 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:30.067144    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3962933b-4378-4941-8983-d4ca1b616be1-istio-data\") pod \"3962933b-4378-4941-8983-d4ca1b616be1\" (UID: \"3962933b-4378-4941-8983-d4ca1b616be1\") "
Sep 14 09:59:30.067347 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:30.067168    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3962933b-4378-4941-8983-d4ca1b616be1-istio-podinfo\") pod \"3962933b-4378-4941-8983-d4ca1b616be1\" (UID: \"3962933b-4378-4941-8983-d4ca1b616be1\") "
Sep 14 09:59:30.067347 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:30.067305    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3962933b-4378-4941-8983-d4ca1b616be1-workload-certs\") pod \"3962933b-4378-4941-8983-d4ca1b616be1\" (UID: \"3962933b-4378-4941-8983-d4ca1b616be1\") "
Sep 14 09:59:30.067552 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:30.067345    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3962933b-4378-4941-8983-d4ca1b616be1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3962933b-4378-4941-8983-d4ca1b616be1" (UID: "3962933b-4378-4941-8983-d4ca1b616be1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 09:59:30.067552 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:30.067354    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lnhz8\" (UniqueName: \"kubernetes.io/projected/3962933b-4378-4941-8983-d4ca1b616be1-kube-api-access-lnhz8\") pod \"3962933b-4378-4941-8983-d4ca1b616be1\" (UID: \"3962933b-4378-4941-8983-d4ca1b616be1\") "
Sep 14 09:59:30.067762 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:30.067686    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3962933b-4378-4941-8983-d4ca1b616be1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3962933b-4378-4941-8983-d4ca1b616be1" (UID: "3962933b-4378-4941-8983-d4ca1b616be1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 09:59:30.067762 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:30.067699    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3962933b-4378-4941-8983-d4ca1b616be1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3962933b-4378-4941-8983-d4ca1b616be1" (UID: "3962933b-4378-4941-8983-d4ca1b616be1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 09:59:30.067965 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:30.067817    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3962933b-4378-4941-8983-d4ca1b616be1-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:59:30.067965 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:30.067838    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3962933b-4378-4941-8983-d4ca1b616be1-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:59:30.067965 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:30.067847    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3962933b-4378-4941-8983-d4ca1b616be1-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:59:30.067965 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:30.067818    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3962933b-4378-4941-8983-d4ca1b616be1-istio-data" (OuterVolumeSpecName: "istio-data") pod "3962933b-4378-4941-8983-d4ca1b616be1" (UID: "3962933b-4378-4941-8983-d4ca1b616be1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 09:59:30.069585 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:30.069552    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3962933b-4378-4941-8983-d4ca1b616be1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3962933b-4378-4941-8983-d4ca1b616be1" (UID: "3962933b-4378-4941-8983-d4ca1b616be1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 09:59:30.069925 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:30.069907    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3962933b-4378-4941-8983-d4ca1b616be1-istio-token" (OuterVolumeSpecName: "istio-token") pod "3962933b-4378-4941-8983-d4ca1b616be1" (UID: "3962933b-4378-4941-8983-d4ca1b616be1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 09:59:30.070036 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:30.069935    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3962933b-4378-4941-8983-d4ca1b616be1-kube-api-access-lnhz8" (OuterVolumeSpecName: "kube-api-access-lnhz8") pod "3962933b-4378-4941-8983-d4ca1b616be1" (UID: "3962933b-4378-4941-8983-d4ca1b616be1"). InnerVolumeSpecName "kube-api-access-lnhz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 09:59:30.168949 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:30.168906    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3962933b-4378-4941-8983-d4ca1b616be1-workload-socket\") pod \"3962933b-4378-4941-8983-d4ca1b616be1\" (UID: \"3962933b-4378-4941-8983-d4ca1b616be1\") "
Sep 14 09:59:30.169178 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:30.168963    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3962933b-4378-4941-8983-d4ca1b616be1-istio-envoy\") pod \"3962933b-4378-4941-8983-d4ca1b616be1\" (UID: \"3962933b-4378-4941-8983-d4ca1b616be1\") "
Sep 14 09:59:30.169266 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:30.169241    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3962933b-4378-4941-8983-d4ca1b616be1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3962933b-4378-4941-8983-d4ca1b616be1" (UID: "3962933b-4378-4941-8983-d4ca1b616be1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 09:59:30.169303 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:30.169265    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3962933b-4378-4941-8983-d4ca1b616be1-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:59:30.169303 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:30.169278    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3962933b-4378-4941-8983-d4ca1b616be1-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:59:30.169303 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:30.169286    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3962933b-4378-4941-8983-d4ca1b616be1-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:59:30.169303 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:30.169294    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnhz8\" (UniqueName: \"kubernetes.io/projected/3962933b-4378-4941-8983-d4ca1b616be1-kube-api-access-lnhz8\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:59:30.171176 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:30.171143    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3962933b-4378-4941-8983-d4ca1b616be1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3962933b-4378-4941-8983-d4ca1b616be1" (UID: "3962933b-4378-4941-8983-d4ca1b616be1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 09:59:30.270729 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:30.270687    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3962933b-4378-4941-8983-d4ca1b616be1-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:59:30.270729 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:30.270722    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3962933b-4378-4941-8983-d4ca1b616be1-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:59:30.726826 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:30.726701    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:59:30.836390 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:30.836352    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:59:30.836748 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:30.836728    2563 generic.go:356] "Generic (PLEG): container finished" podID="3962933b-4378-4941-8983-d4ca1b616be1" containerID="2432695cc18bedcf009a74c87d0fd83d1dc509ffad962c4bcad1ebb3d11e3ee7" exitCode=0
Sep 14 09:59:30.836816 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:30.836777    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl"
Sep 14 09:59:30.836852 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:30.836805    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl" event={"ID":"3962933b-4378-4941-8983-d4ca1b616be1","Type":"ContainerDied","Data":"2432695cc18bedcf009a74c87d0fd83d1dc509ffad962c4bcad1ebb3d11e3ee7"}
Sep 14 09:59:30.836895 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:30.836865    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl" event={"ID":"3962933b-4378-4941-8983-d4ca1b616be1","Type":"ContainerDied","Data":"09ddd8b90bd5c9b345a0732cc7aca0a28664149738a8c6cfc7252d7565d3b49c"}
Sep 14 09:59:30.836928 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:30.836897    2563 scope.go:117] "RemoveContainer" containerID="2432695cc18bedcf009a74c87d0fd83d1dc509ffad962c4bcad1ebb3d11e3ee7"
Sep 14 09:59:30.847304 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:30.847284    2563 scope.go:117] "RemoveContainer" containerID="ef3cd82a4ff7f6a430401e9722075656a95984b7949e0ff9b36a513f6fe7fd3d"
Sep 14 09:59:30.857553 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:30.857531    2563 scope.go:117] "RemoveContainer" containerID="2432695cc18bedcf009a74c87d0fd83d1dc509ffad962c4bcad1ebb3d11e3ee7"
Sep 14 09:59:30.857906 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:59:30.857881    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2432695cc18bedcf009a74c87d0fd83d1dc509ffad962c4bcad1ebb3d11e3ee7\": container with ID starting with 2432695cc18bedcf009a74c87d0fd83d1dc509ffad962c4bcad1ebb3d11e3ee7 not found: ID does not exist" containerID="2432695cc18bedcf009a74c87d0fd83d1dc509ffad962c4bcad1ebb3d11e3ee7"
Sep 14 09:59:30.857995 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:30.857916    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2432695cc18bedcf009a74c87d0fd83d1dc509ffad962c4bcad1ebb3d11e3ee7"} err="failed to get container status \"2432695cc18bedcf009a74c87d0fd83d1dc509ffad962c4bcad1ebb3d11e3ee7\": rpc error: code = NotFound desc = could not find container \"2432695cc18bedcf009a74c87d0fd83d1dc509ffad962c4bcad1ebb3d11e3ee7\": container with ID starting with 2432695cc18bedcf009a74c87d0fd83d1dc509ffad962c4bcad1ebb3d11e3ee7 not found: ID does not exist"
Sep 14 09:59:30.857995 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:30.857937    2563 scope.go:117] "RemoveContainer" containerID="ef3cd82a4ff7f6a430401e9722075656a95984b7949e0ff9b36a513f6fe7fd3d"
Sep 14 09:59:30.858272 ip-10-0-132-4 kubenswrapper[2563]: E0914 09:59:30.858252    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef3cd82a4ff7f6a430401e9722075656a95984b7949e0ff9b36a513f6fe7fd3d\": container with ID starting with ef3cd82a4ff7f6a430401e9722075656a95984b7949e0ff9b36a513f6fe7fd3d not found: ID does not exist" containerID="ef3cd82a4ff7f6a430401e9722075656a95984b7949e0ff9b36a513f6fe7fd3d"
Sep 14 09:59:30.858341 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:30.858280    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef3cd82a4ff7f6a430401e9722075656a95984b7949e0ff9b36a513f6fe7fd3d"} err="failed to get container status \"ef3cd82a4ff7f6a430401e9722075656a95984b7949e0ff9b36a513f6fe7fd3d\": rpc error: code = NotFound desc = could not find container \"ef3cd82a4ff7f6a430401e9722075656a95984b7949e0ff9b36a513f6fe7fd3d\": container with ID starting with ef3cd82a4ff7f6a430401e9722075656a95984b7949e0ff9b36a513f6fe7fd3d not found: ID does not exist"
Sep 14 09:59:30.866158 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:30.866117    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl"]
Sep 14 09:59:30.874476 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:30.874434    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-e7fe0e18-openshift-default-6dc8f7746b-8pzdl"]
Sep 14 09:59:31.108526 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:31.108493    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:59:31.419610 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:31.419511    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:59:31.484097 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:31.484061    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3962933b-4378-4941-8983-d4ca1b616be1" path="/var/lib/kubelet/pods/3962933b-4378-4941-8983-d4ca1b616be1/volumes"
Sep 14 09:59:31.514592 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:31.514552    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:59:31.905445 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:31.905403    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:59:33.136523 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:33.136481    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:59:34.107069 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:34.107037    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:59:34.946620 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:34.946581    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:59:35.772476 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:35.772442    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:59:53.123104 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:53.123058    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-80fa7537-mqrfv"]
Sep 14 09:59:53.123501 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:53.123482    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3962933b-4378-4941-8983-d4ca1b616be1" containerName="istio-proxy"
Sep 14 09:59:53.123501 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:53.123494    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3962933b-4378-4941-8983-d4ca1b616be1" containerName="istio-proxy"
Sep 14 09:59:53.123575 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:53.123503    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3962933b-4378-4941-8983-d4ca1b616be1" containerName="istio-proxy"
Sep 14 09:59:53.123575 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:53.123510    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3962933b-4378-4941-8983-d4ca1b616be1" containerName="istio-proxy"
Sep 14 09:59:53.123575 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:53.123571    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3962933b-4378-4941-8983-d4ca1b616be1" containerName="istio-proxy"
Sep 14 09:59:53.123674 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:53.123580    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3962933b-4378-4941-8983-d4ca1b616be1" containerName="istio-proxy"
Sep 14 09:59:53.126830 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:53.126808    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-80fa7537-mqrfv"
Sep 14 09:59:53.129316 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:53.129283    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gf4xj\""
Sep 14 09:59:53.129480 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:53.129284    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 14 09:59:53.139642 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:53.139609    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-80fa7537-mqrfv"]
Sep 14 09:59:53.210496 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:53.210452    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8483b2cd-a574-42e8-a49d-35f26cad6599-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-80fa7537-mqrfv\" (UID: \"8483b2cd-a574-42e8-a49d-35f26cad6599\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-80fa7537-mqrfv"
Sep 14 09:59:53.311220 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:53.311119    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8483b2cd-a574-42e8-a49d-35f26cad6599-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-80fa7537-mqrfv\" (UID: \"8483b2cd-a574-42e8-a49d-35f26cad6599\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-80fa7537-mqrfv"
Sep 14 09:59:53.312619 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:53.311776    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8483b2cd-a574-42e8-a49d-35f26cad6599-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-80fa7537-mqrfv\" (UID: \"8483b2cd-a574-42e8-a49d-35f26cad6599\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-80fa7537-mqrfv"
Sep 14 09:59:53.436638 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:53.436597    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:59:53.440504 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:53.440475    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-80fa7537-mqrfv"
Sep 14 09:59:53.733773 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:53.733740    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-80fa7537-mqrfv"]
Sep 14 09:59:53.734832 ip-10-0-132-4 kubenswrapper[2563]: W0914 09:59:53.734809    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8483b2cd_a574_42e8_a49d_35f26cad6599.slice/crio-682d0d53f6f670dc400ed64b302c8b5accbef054cd46a5c1c001b289e980d875 WatchSource:0}: Error finding container 682d0d53f6f670dc400ed64b302c8b5accbef054cd46a5c1c001b289e980d875: Status 404 returned error can't find the container with id 682d0d53f6f670dc400ed64b302c8b5accbef054cd46a5c1c001b289e980d875
Sep 14 09:59:54.000263 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:54.000227    2563 generic.go:356] "Generic (PLEG): container finished" podID="309ac5a4-3502-4351-830d-e6cf50ab73be" containerID="f9fabeae6867291a9e9ca5da8d5aef251af47ecb90ac1fe8f7c7de5619ff90a3" exitCode=0
Sep 14 09:59:54.000452 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:54.000302    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm" event={"ID":"309ac5a4-3502-4351-830d-e6cf50ab73be","Type":"ContainerDied","Data":"f9fabeae6867291a9e9ca5da8d5aef251af47ecb90ac1fe8f7c7de5619ff90a3"}
Sep 14 09:59:54.000849 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:54.000826    2563 scope.go:117] "RemoveContainer" containerID="f9fabeae6867291a9e9ca5da8d5aef251af47ecb90ac1fe8f7c7de5619ff90a3"
Sep 14 09:59:54.001888 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:54.001865    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-80fa7537-mqrfv" event={"ID":"8483b2cd-a574-42e8-a49d-35f26cad6599","Type":"ContainerStarted","Data":"682d0d53f6f670dc400ed64b302c8b5accbef054cd46a5c1c001b289e980d875"}
Sep 14 09:59:54.002106 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:54.002081    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 09:59:54.002162 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:54.002149    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 09:59:54.002203 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:54.002180    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 09:59:54.411757 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:54.411624    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:59:55.009358 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:55.009318    2563 generic.go:356] "Generic (PLEG): container finished" podID="8483b2cd-a574-42e8-a49d-35f26cad6599" containerID="ce2d1a938076f6e45916cfe1d9dd46da4c8fb0cab318181af11d18a37cefa812" exitCode=0
Sep 14 09:59:55.009558 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:55.009407    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-80fa7537-mqrfv" event={"ID":"8483b2cd-a574-42e8-a49d-35f26cad6599","Type":"ContainerDied","Data":"ce2d1a938076f6e45916cfe1d9dd46da4c8fb0cab318181af11d18a37cefa812"}
Sep 14 09:59:55.011675 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:55.011650    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm" event={"ID":"309ac5a4-3502-4351-830d-e6cf50ab73be","Type":"ContainerStarted","Data":"560183e951ece993d34ee8f1436b82ad8f228500716d64a592daea81e3df3724"}
Sep 14 09:59:55.413799 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:55.413712    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:59:55.872150 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:55.872110    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm"
Sep 14 09:59:55.873455 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:55.873424    2563 patch_prober.go:28] interesting pod/e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Sep 14 09:59:55.873571 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:55.873487    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm" podUID="309ac5a4-3502-4351-830d-e6cf50ab73be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Sep 14 09:59:56.199051 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:56.199016    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-80fa7537-mqrfv"
Sep 14 09:59:56.354139 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:56.354099    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8483b2cd-a574-42e8-a49d-35f26cad6599-maas-api-service-ca\") pod \"8483b2cd-a574-42e8-a49d-35f26cad6599\" (UID: \"8483b2cd-a574-42e8-a49d-35f26cad6599\") "
Sep 14 09:59:56.354483 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:56.354458    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8483b2cd-a574-42e8-a49d-35f26cad6599-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8483b2cd-a574-42e8-a49d-35f26cad6599" (UID: "8483b2cd-a574-42e8-a49d-35f26cad6599"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 09:59:56.414413 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:56.414374    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:59:56.455228 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:56.455141    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8483b2cd-a574-42e8-a49d-35f26cad6599-maas-api-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 09:59:56.872712 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:56.872672    2563 patch_prober.go:28] interesting pod/e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Sep 14 09:59:56.872908 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:56.872732    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm" podUID="309ac5a4-3502-4351-830d-e6cf50ab73be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Sep 14 09:59:57.028706 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:57.028668    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-80fa7537-mqrfv"
Sep 14 09:59:57.028930 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:57.028783    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-80fa7537-mqrfv" event={"ID":"8483b2cd-a574-42e8-a49d-35f26cad6599","Type":"ContainerDied","Data":"682d0d53f6f670dc400ed64b302c8b5accbef054cd46a5c1c001b289e980d875"}
Sep 14 09:59:57.028930 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:57.028829    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="682d0d53f6f670dc400ed64b302c8b5accbef054cd46a5c1c001b289e980d875"
Sep 14 09:59:57.220049 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:57.219942    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:59:57.872200 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:57.872147    2563 patch_prober.go:28] interesting pod/e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Sep 14 09:59:57.872709 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:57.872228    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm" podUID="309ac5a4-3502-4351-830d-e6cf50ab73be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Sep 14 09:59:58.047484 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:58.047443    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:59:58.796265 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:58.796234    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:59:58.822410 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:58.822356    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm"
Sep 14 09:59:58.872345 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:58.872304    2563 patch_prober.go:28] interesting pod/e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Sep 14 09:59:58.872756 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:58.872375    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm" podUID="309ac5a4-3502-4351-830d-e6cf50ab73be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Sep 14 09:59:59.692777 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:59.692736    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 09:59:59.872885 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:59.872845    2563 patch_prober.go:28] interesting pod/e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Sep 14 09:59:59.873347 ip-10-0-132-4 kubenswrapper[2563]: I0914 09:59:59.872917    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm" podUID="309ac5a4-3502-4351-830d-e6cf50ab73be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Sep 14 10:00:00.158069 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:00.158020    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-8c98aa-29823000-6dbrt"]
Sep 14 10:00:00.158463 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:00.158449    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8483b2cd-a574-42e8-a49d-35f26cad6599" containerName="revoke-keys"
Sep 14 10:00:00.158520 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:00.158465    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8483b2cd-a574-42e8-a49d-35f26cad6599" containerName="revoke-keys"
Sep 14 10:00:00.158559 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:00.158525    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8483b2cd-a574-42e8-a49d-35f26cad6599" containerName="revoke-keys"
Sep 14 10:00:00.160780 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:00.160760    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-8c98aa-29823000-6dbrt"
Sep 14 10:00:00.164184 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:00.164145    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gf4xj\""
Sep 14 10:00:00.171928 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:00.171892    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-8c98aa-29823000-6dbrt"
Sep 14 10:00:00.175229 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:00.175178    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-8c98aa-29823000-6dbrt"]
Sep 14 10:00:00.178876 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:00.178830    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6812e-29823000-g4fqs"]
Sep 14 10:00:00.182049 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:00.182021    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6812e-29823000-g4fqs"
Sep 14 10:00:00.183081 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:00.183051    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0e60f3-29823000-9r5gg"]
Sep 14 10:00:00.186091 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:00.186067    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0e60f3-29823000-9r5gg"
Sep 14 10:00:00.186249 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:00.186144    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29823000-b58d6"]
Sep 14 10:00:00.188767 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:00.188739    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29823000-b58d6"
Sep 14 10:00:00.200875 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:00.200838    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6812e-29823000-g4fqs"]
Sep 14 10:00:00.223921 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:00.223683    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6812e-29823000-g4fqs"
Sep 14 10:00:00.260807 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:00.260776    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0e60f3-29823000-9r5gg"
Sep 14 10:00:00.277291 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:00.275096    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0e60f3-29823000-9r5gg"]
Sep 14 10:00:00.298701 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:00.298484    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2901ec-29823000-mn5kc"]
Sep 14 10:00:00.305993 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:00.305953    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2901ec-29823000-mn5kc"
Sep 14 10:00:00.317346 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:00.317281    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29823000-b58d6"]
Sep 14 10:00:00.341835 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:00.341092    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2901ec-29823000-mn5kc"]
Sep 14 10:00:00.354050 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:00.353094    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29823000-b58d6"
Sep 14 10:00:00.377164 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:00.377132    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2901ec-29823000-mn5kc"
Sep 14 10:00:00.551026 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:00.549633    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:00:00.575221 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:00.575171    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-8c98aa-29823000-6dbrt"]
Sep 14 10:00:00.577737 ip-10-0-132-4 kubenswrapper[2563]: W0914 10:00:00.577551    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod116b063c_bcb9_4a35_91c6_4c894aec8d4a.slice/crio-96aa4dc808b4e0d60f6e224d76aa982bfd6a6a99e4fc2547d32b2f1c0d0349b9 WatchSource:0}: Error finding container 96aa4dc808b4e0d60f6e224d76aa982bfd6a6a99e4fc2547d32b2f1c0d0349b9: Status 404 returned error can't find the container with id 96aa4dc808b4e0d60f6e224d76aa982bfd6a6a99e4fc2547d32b2f1c0d0349b9
Sep 14 10:00:00.650784 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:00.650736    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6812e-29823000-g4fqs"]
Sep 14 10:00:00.664992 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:00.664944    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0e60f3-29823000-9r5gg"]
Sep 14 10:00:00.665404 ip-10-0-132-4 kubenswrapper[2563]: W0914 10:00:00.665332    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45f8f6ed_1ae8_4268_9224_6c58faa23fee.slice/crio-016698a4ae4f664152ef50a8e10e39834128dedecbc3c5419ac25655d79d6307 WatchSource:0}: Error finding container 016698a4ae4f664152ef50a8e10e39834128dedecbc3c5419ac25655d79d6307: Status 404 returned error can't find the container with id 016698a4ae4f664152ef50a8e10e39834128dedecbc3c5419ac25655d79d6307
Sep 14 10:00:00.671289 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:00.671176    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:00:00.702501 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:00.702474    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29823000-b58d6"]
Sep 14 10:00:00.704739 ip-10-0-132-4 kubenswrapper[2563]: W0914 10:00:00.704714    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f9f3e6b_0583_421c_90ad_c83e44e60580.slice/crio-60492c20911117351ee889c48d9ef9378cf5de90ba9904abbff2415b7e61d712 WatchSource:0}: Error finding container 60492c20911117351ee889c48d9ef9378cf5de90ba9904abbff2415b7e61d712: Status 404 returned error can't find the container with id 60492c20911117351ee889c48d9ef9378cf5de90ba9904abbff2415b7e61d712
Sep 14 10:00:00.717916 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:00.717870    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2901ec-29823000-mn5kc"]
Sep 14 10:00:00.720426 ip-10-0-132-4 kubenswrapper[2563]: W0914 10:00:00.720388    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod873f88a1_9881_41e8_8bc2_cff73ebbfed7.slice/crio-9fe5112d688da383542f870261940affb67791bad9f42f40e29a1e2d61d36083 WatchSource:0}: Error finding container 9fe5112d688da383542f870261940affb67791bad9f42f40e29a1e2d61d36083: Status 404 returned error can't find the container with id 9fe5112d688da383542f870261940affb67791bad9f42f40e29a1e2d61d36083
Sep 14 10:00:00.764670 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:00.764623    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:00:00.879980 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:00.879925    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:00:00.929937 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:00.929832    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-80fa7537-mqrfv"]
Sep 14 10:00:00.941322 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:00.941273    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-80fa7537-mqrfv"]
Sep 14 10:00:00.994255 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:00.994218    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:00:01.075632 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:01.075590    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2901ec-29823000-mn5kc" event={"ID":"873f88a1-9881-41e8-8bc2-cff73ebbfed7","Type":"ContainerStarted","Data":"9fe5112d688da383542f870261940affb67791bad9f42f40e29a1e2d61d36083"}
Sep 14 10:00:01.078862 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:01.078783    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29823000-b58d6" event={"ID":"3f9f3e6b-0583-421c-90ad-c83e44e60580","Type":"ContainerStarted","Data":"60492c20911117351ee889c48d9ef9378cf5de90ba9904abbff2415b7e61d712"}
Sep 14 10:00:01.082391 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:01.081744    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-8c98aa-29823000-6dbrt" event={"ID":"116b063c-bcb9-4a35-91c6-4c894aec8d4a","Type":"ContainerStarted","Data":"0c35c8a327c3b23125bbdb1bc4f199afbc2cf72633c32d105b5714f6101d068f"}
Sep 14 10:00:01.082391 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:01.081826    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-8c98aa-29823000-6dbrt" event={"ID":"116b063c-bcb9-4a35-91c6-4c894aec8d4a","Type":"ContainerStarted","Data":"96aa4dc808b4e0d60f6e224d76aa982bfd6a6a99e4fc2547d32b2f1c0d0349b9"}
Sep 14 10:00:01.083230 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:01.083191    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm"]
Sep 14 10:00:01.083649 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:01.083615    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm" podUID="309ac5a4-3502-4351-830d-e6cf50ab73be" containerName="istio-proxy" containerID="cri-o://560183e951ece993d34ee8f1436b82ad8f228500716d64a592daea81e3df3724" gracePeriod=30
Sep 14 10:00:01.085154 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:01.085123    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0e60f3-29823000-9r5gg" event={"ID":"45f8f6ed-1ae8-4268-9224-6c58faa23fee","Type":"ContainerStarted","Data":"81530ebfa06370b4e74fdefb5fb6d9acb28b651ba4c15d4fa80ec6bbf74d1708"}
Sep 14 10:00:01.085568 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:01.085184    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0e60f3-29823000-9r5gg" event={"ID":"45f8f6ed-1ae8-4268-9224-6c58faa23fee","Type":"ContainerStarted","Data":"016698a4ae4f664152ef50a8e10e39834128dedecbc3c5419ac25655d79d6307"}
Sep 14 10:00:01.086928 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:01.086893    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6812e-29823000-g4fqs" event={"ID":"7d123f30-e7e7-4a5a-add5-73ee74bf4742","Type":"ContainerStarted","Data":"83b5c131c198abb53a77cb5d32adb99d661b768a83984630f50531778c25b884"}
Sep 14 10:00:01.100712 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:01.100673    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:00:01.103147 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:01.103084    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-8c98aa-29823000-6dbrt" podStartSLOduration=1.103060854 podStartE2EDuration="1.103060854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:00:01.09895589 +0000 UTC m=+990.138630513" watchObservedRunningTime="2026-09-14 10:00:01.103060854 +0000 UTC m=+990.142735476"
Sep 14 10:00:01.122981 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:01.122937    2563 patch_prober.go:28] interesting pod/e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 14 10:00:01.123183 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:01.123022    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm" podUID="309ac5a4-3502-4351-830d-e6cf50ab73be" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 14 10:00:01.141672 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:01.141616    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0e60f3-29823000-9r5gg" podStartSLOduration=1.141597025 podStartE2EDuration="1.141597025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:00:01.140086256 +0000 UTC m=+990.179760877" watchObservedRunningTime="2026-09-14 10:00:01.141597025 +0000 UTC m=+990.181271637"
Sep 14 10:00:01.486792 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:01.486751    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8483b2cd-a574-42e8-a49d-35f26cad6599" path="/var/lib/kubelet/pods/8483b2cd-a574-42e8-a49d-35f26cad6599/volumes"
Sep 14 10:00:01.804437 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:01.804336    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:00:02.097481 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:02.097375    2563 generic.go:356] "Generic (PLEG): container finished" podID="7d123f30-e7e7-4a5a-add5-73ee74bf4742" containerID="341d49e2e204ae07b199775216732718362e567a846d49d95e0005b9ee7e00cb" exitCode=0
Sep 14 10:00:02.097481 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:02.097460    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6812e-29823000-g4fqs" event={"ID":"7d123f30-e7e7-4a5a-add5-73ee74bf4742","Type":"ContainerDied","Data":"341d49e2e204ae07b199775216732718362e567a846d49d95e0005b9ee7e00cb"}
Sep 14 10:00:02.099309 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:02.099277    2563 generic.go:356] "Generic (PLEG): container finished" podID="873f88a1-9881-41e8-8bc2-cff73ebbfed7" containerID="42d6b78d83b789ba35bf2ffee8b1e2e5b374748a723ca5f503ddec02a546a4ee" exitCode=0
Sep 14 10:00:02.099446 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:02.099364    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2901ec-29823000-mn5kc" event={"ID":"873f88a1-9881-41e8-8bc2-cff73ebbfed7","Type":"ContainerDied","Data":"42d6b78d83b789ba35bf2ffee8b1e2e5b374748a723ca5f503ddec02a546a4ee"}
Sep 14 10:00:02.102089 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:02.102054    2563 generic.go:356] "Generic (PLEG): container finished" podID="3f9f3e6b-0583-421c-90ad-c83e44e60580" containerID="3af18e9b9660e57cb9c22fad907de226e159fb4dfe36e935fb04851ff1cb56d5" exitCode=0
Sep 14 10:00:02.102241 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:02.102134    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29823000-b58d6" event={"ID":"3f9f3e6b-0583-421c-90ad-c83e44e60580","Type":"ContainerDied","Data":"3af18e9b9660e57cb9c22fad907de226e159fb4dfe36e935fb04851ff1cb56d5"}
Sep 14 10:00:02.103849 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:02.103825    2563 generic.go:356] "Generic (PLEG): container finished" podID="116b063c-bcb9-4a35-91c6-4c894aec8d4a" containerID="0c35c8a327c3b23125bbdb1bc4f199afbc2cf72633c32d105b5714f6101d068f" exitCode=0
Sep 14 10:00:02.103991 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:02.103909    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-8c98aa-29823000-6dbrt" event={"ID":"116b063c-bcb9-4a35-91c6-4c894aec8d4a","Type":"ContainerDied","Data":"0c35c8a327c3b23125bbdb1bc4f199afbc2cf72633c32d105b5714f6101d068f"}
Sep 14 10:00:02.105587 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:02.105556    2563 generic.go:356] "Generic (PLEG): container finished" podID="45f8f6ed-1ae8-4268-9224-6c58faa23fee" containerID="81530ebfa06370b4e74fdefb5fb6d9acb28b651ba4c15d4fa80ec6bbf74d1708" exitCode=0
Sep 14 10:00:02.105712 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:02.105600    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0e60f3-29823000-9r5gg" event={"ID":"45f8f6ed-1ae8-4268-9224-6c58faa23fee","Type":"ContainerDied","Data":"81530ebfa06370b4e74fdefb5fb6d9acb28b651ba4c15d4fa80ec6bbf74d1708"}
Sep 14 10:00:02.125359 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:02.125321    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:00:02.768447 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:02.768410    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:00:03.291162 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:03.290656    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6812e-29823000-g4fqs"
Sep 14 10:00:03.348683 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:03.348651    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29823000-b58d6"
Sep 14 10:00:03.352629 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:03.352601    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2901ec-29823000-mn5kc"
Sep 14 10:00:03.357426 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:03.357398    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0e60f3-29823000-9r5gg"
Sep 14 10:00:03.361134 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:03.361104    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-8c98aa-29823000-6dbrt"
Sep 14 10:00:03.548797 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:03.548756    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:00:03.746602 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:03.746555    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:00:04.121343 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:04.121312    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-8c98aa-29823000-6dbrt"
Sep 14 10:00:04.121542 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:04.121340    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-8c98aa-29823000-6dbrt" event={"ID":"116b063c-bcb9-4a35-91c6-4c894aec8d4a","Type":"ContainerDied","Data":"96aa4dc808b4e0d60f6e224d76aa982bfd6a6a99e4fc2547d32b2f1c0d0349b9"}
Sep 14 10:00:04.121542 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:04.121395    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96aa4dc808b4e0d60f6e224d76aa982bfd6a6a99e4fc2547d32b2f1c0d0349b9"
Sep 14 10:00:04.122833 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:04.122810    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0e60f3-29823000-9r5gg"
Sep 14 10:00:04.122982 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:04.122843    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0e60f3-29823000-9r5gg" event={"ID":"45f8f6ed-1ae8-4268-9224-6c58faa23fee","Type":"ContainerDied","Data":"016698a4ae4f664152ef50a8e10e39834128dedecbc3c5419ac25655d79d6307"}
Sep 14 10:00:04.122982 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:04.122886    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="016698a4ae4f664152ef50a8e10e39834128dedecbc3c5419ac25655d79d6307"
Sep 14 10:00:04.124608 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:04.124582    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6812e-29823000-g4fqs"
Sep 14 10:00:04.124769 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:04.124611    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6812e-29823000-g4fqs" event={"ID":"7d123f30-e7e7-4a5a-add5-73ee74bf4742","Type":"ContainerDied","Data":"83b5c131c198abb53a77cb5d32adb99d661b768a83984630f50531778c25b884"}
Sep 14 10:00:04.124769 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:04.124662    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83b5c131c198abb53a77cb5d32adb99d661b768a83984630f50531778c25b884"
Sep 14 10:00:04.126314 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:04.126294    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2901ec-29823000-mn5kc"
Sep 14 10:00:04.126460 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:04.126291    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2901ec-29823000-mn5kc" event={"ID":"873f88a1-9881-41e8-8bc2-cff73ebbfed7","Type":"ContainerDied","Data":"9fe5112d688da383542f870261940affb67791bad9f42f40e29a1e2d61d36083"}
Sep 14 10:00:04.126460 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:04.126422    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9fe5112d688da383542f870261940affb67791bad9f42f40e29a1e2d61d36083"
Sep 14 10:00:04.127841 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:04.127811    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29823000-b58d6" event={"ID":"3f9f3e6b-0583-421c-90ad-c83e44e60580","Type":"ContainerDied","Data":"60492c20911117351ee889c48d9ef9378cf5de90ba9904abbff2415b7e61d712"}
Sep 14 10:00:04.127971 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:04.127842    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29823000-b58d6"
Sep 14 10:00:04.127971 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:04.127863    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60492c20911117351ee889c48d9ef9378cf5de90ba9904abbff2415b7e61d712"
Sep 14 10:00:04.804416 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:04.804376    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:00:05.202701 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:05.202598    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:00:05.460886 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:05.460777    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-0e60f3/test-model-0e60f3-kserve-84c9545bb9-bx4l2"]
Sep 14 10:00:05.461277 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:05.461259    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f9f3e6b-0583-421c-90ad-c83e44e60580" containerName="cleanup"
Sep 14 10:00:05.461341 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:05.461279    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f9f3e6b-0583-421c-90ad-c83e44e60580" containerName="cleanup"
Sep 14 10:00:05.461341 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:05.461299    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45f8f6ed-1ae8-4268-9224-6c58faa23fee" containerName="cleanup"
Sep 14 10:00:05.461341 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:05.461305    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="45f8f6ed-1ae8-4268-9224-6c58faa23fee" containerName="cleanup"
Sep 14 10:00:05.461341 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:05.461311    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="873f88a1-9881-41e8-8bc2-cff73ebbfed7" containerName="cleanup"
Sep 14 10:00:05.461341 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:05.461317    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="873f88a1-9881-41e8-8bc2-cff73ebbfed7" containerName="cleanup"
Sep 14 10:00:05.461341 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:05.461327    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d123f30-e7e7-4a5a-add5-73ee74bf4742" containerName="cleanup"
Sep 14 10:00:05.461341 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:05.461332    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d123f30-e7e7-4a5a-add5-73ee74bf4742" containerName="cleanup"
Sep 14 10:00:05.461341 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:05.461342    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="116b063c-bcb9-4a35-91c6-4c894aec8d4a" containerName="cleanup"
Sep 14 10:00:05.461581 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:05.461347    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="116b063c-bcb9-4a35-91c6-4c894aec8d4a" containerName="cleanup"
Sep 14 10:00:05.461581 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:05.461412    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="45f8f6ed-1ae8-4268-9224-6c58faa23fee" containerName="cleanup"
Sep 14 10:00:05.461581 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:05.461421    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="116b063c-bcb9-4a35-91c6-4c894aec8d4a" containerName="cleanup"
Sep 14 10:00:05.461581 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:05.461427    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d123f30-e7e7-4a5a-add5-73ee74bf4742" containerName="cleanup"
Sep 14 10:00:05.461581 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:05.461433    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f9f3e6b-0583-421c-90ad-c83e44e60580" containerName="cleanup"
Sep 14 10:00:05.461581 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:05.461439    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="873f88a1-9881-41e8-8bc2-cff73ebbfed7" containerName="cleanup"
Sep 14 10:00:05.463593 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:05.463564    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0e60f3/test-model-0e60f3-kserve-84c9545bb9-bx4l2"
Sep 14 10:00:05.466042 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:05.466017    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-0e60f3\"/\"openshift-service-ca.crt\""
Sep 14 10:00:05.466686 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:05.466660    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-0e60f3\"/\"kube-root-ca.crt\""
Sep 14 10:00:05.466784 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:05.466696    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-0e60f3\"/\"test-model-0e60f3-kserve-self-signed-certs\""
Sep 14 10:00:05.466784 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:05.466707    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-0e60f3\"/\"default-dockercfg-8g72c\""
Sep 14 10:00:05.483505 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:05.483473    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-0e60f3/test-model-0e60f3-kserve-84c9545bb9-bx4l2"]
Sep 14 10:00:05.551389 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:05.551346    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90443638-9dd8-4c4d-812f-146cba869af7-dshm\") pod \"test-model-0e60f3-kserve-84c9545bb9-bx4l2\" (UID: \"90443638-9dd8-4c4d-812f-146cba869af7\") " pod="ai-tenant-e2e-inf-a-0e60f3/test-model-0e60f3-kserve-84c9545bb9-bx4l2"
Sep 14 10:00:05.551389 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:05.551385    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90443638-9dd8-4c4d-812f-146cba869af7-model-cache\") pod \"test-model-0e60f3-kserve-84c9545bb9-bx4l2\" (UID: \"90443638-9dd8-4c4d-812f-146cba869af7\") " pod="ai-tenant-e2e-inf-a-0e60f3/test-model-0e60f3-kserve-84c9545bb9-bx4l2"
Sep 14 10:00:05.551623 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:05.551520    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90443638-9dd8-4c4d-812f-146cba869af7-tmp-dir\") pod \"test-model-0e60f3-kserve-84c9545bb9-bx4l2\" (UID: \"90443638-9dd8-4c4d-812f-146cba869af7\") " pod="ai-tenant-e2e-inf-a-0e60f3/test-model-0e60f3-kserve-84c9545bb9-bx4l2"
Sep 14 10:00:05.551623 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:05.551549    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90443638-9dd8-4c4d-812f-146cba869af7-tls-certs\") pod \"test-model-0e60f3-kserve-84c9545bb9-bx4l2\" (UID: \"90443638-9dd8-4c4d-812f-146cba869af7\") " pod="ai-tenant-e2e-inf-a-0e60f3/test-model-0e60f3-kserve-84c9545bb9-bx4l2"
Sep 14 10:00:05.551623 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:05.551588    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qvrj\" (UniqueName: \"kubernetes.io/projected/90443638-9dd8-4c4d-812f-146cba869af7-kube-api-access-7qvrj\") pod \"test-model-0e60f3-kserve-84c9545bb9-bx4l2\" (UID: \"90443638-9dd8-4c4d-812f-146cba869af7\") " pod="ai-tenant-e2e-inf-a-0e60f3/test-model-0e60f3-kserve-84c9545bb9-bx4l2"
Sep 14 10:00:05.551750 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:05.551626    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90443638-9dd8-4c4d-812f-146cba869af7-home\") pod \"test-model-0e60f3-kserve-84c9545bb9-bx4l2\" (UID: \"90443638-9dd8-4c4d-812f-146cba869af7\") " pod="ai-tenant-e2e-inf-a-0e60f3/test-model-0e60f3-kserve-84c9545bb9-bx4l2"
Sep 14 10:00:05.653120 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:05.653077    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90443638-9dd8-4c4d-812f-146cba869af7-tmp-dir\") pod \"test-model-0e60f3-kserve-84c9545bb9-bx4l2\" (UID: \"90443638-9dd8-4c4d-812f-146cba869af7\") " pod="ai-tenant-e2e-inf-a-0e60f3/test-model-0e60f3-kserve-84c9545bb9-bx4l2"
Sep 14 10:00:05.653877 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:05.653852    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90443638-9dd8-4c4d-812f-146cba869af7-tls-certs\") pod \"test-model-0e60f3-kserve-84c9545bb9-bx4l2\" (UID: \"90443638-9dd8-4c4d-812f-146cba869af7\") " pod="ai-tenant-e2e-inf-a-0e60f3/test-model-0e60f3-kserve-84c9545bb9-bx4l2"
Sep 14 10:00:05.654644 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:05.654621    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qvrj\" (UniqueName: \"kubernetes.io/projected/90443638-9dd8-4c4d-812f-146cba869af7-kube-api-access-7qvrj\") pod \"test-model-0e60f3-kserve-84c9545bb9-bx4l2\" (UID: \"90443638-9dd8-4c4d-812f-146cba869af7\") " pod="ai-tenant-e2e-inf-a-0e60f3/test-model-0e60f3-kserve-84c9545bb9-bx4l2"
Sep 14 10:00:05.654812 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:05.654795    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90443638-9dd8-4c4d-812f-146cba869af7-home\") pod \"test-model-0e60f3-kserve-84c9545bb9-bx4l2\" (UID: \"90443638-9dd8-4c4d-812f-146cba869af7\") " pod="ai-tenant-e2e-inf-a-0e60f3/test-model-0e60f3-kserve-84c9545bb9-bx4l2"
Sep 14 10:00:05.655082 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:05.655058    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90443638-9dd8-4c4d-812f-146cba869af7-dshm\") pod \"test-model-0e60f3-kserve-84c9545bb9-bx4l2\" (UID: \"90443638-9dd8-4c4d-812f-146cba869af7\") " pod="ai-tenant-e2e-inf-a-0e60f3/test-model-0e60f3-kserve-84c9545bb9-bx4l2"
Sep 14 10:00:05.655243 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:05.655228    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90443638-9dd8-4c4d-812f-146cba869af7-model-cache\") pod \"test-model-0e60f3-kserve-84c9545bb9-bx4l2\" (UID: \"90443638-9dd8-4c4d-812f-146cba869af7\") " pod="ai-tenant-e2e-inf-a-0e60f3/test-model-0e60f3-kserve-84c9545bb9-bx4l2"
Sep 14 10:00:05.655711 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:05.655687    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90443638-9dd8-4c4d-812f-146cba869af7-model-cache\") pod \"test-model-0e60f3-kserve-84c9545bb9-bx4l2\" (UID: \"90443638-9dd8-4c4d-812f-146cba869af7\") " pod="ai-tenant-e2e-inf-a-0e60f3/test-model-0e60f3-kserve-84c9545bb9-bx4l2"
Sep 14 10:00:05.656116 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:05.656086    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90443638-9dd8-4c4d-812f-146cba869af7-home\") pod \"test-model-0e60f3-kserve-84c9545bb9-bx4l2\" (UID: \"90443638-9dd8-4c4d-812f-146cba869af7\") " pod="ai-tenant-e2e-inf-a-0e60f3/test-model-0e60f3-kserve-84c9545bb9-bx4l2"
Sep 14 10:00:05.656256 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:05.653786    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90443638-9dd8-4c4d-812f-146cba869af7-tmp-dir\") pod \"test-model-0e60f3-kserve-84c9545bb9-bx4l2\" (UID: \"90443638-9dd8-4c4d-812f-146cba869af7\") " pod="ai-tenant-e2e-inf-a-0e60f3/test-model-0e60f3-kserve-84c9545bb9-bx4l2"
Sep 14 10:00:05.659061 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:05.658991    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90443638-9dd8-4c4d-812f-146cba869af7-dshm\") pod \"test-model-0e60f3-kserve-84c9545bb9-bx4l2\" (UID: \"90443638-9dd8-4c4d-812f-146cba869af7\") " pod="ai-tenant-e2e-inf-a-0e60f3/test-model-0e60f3-kserve-84c9545bb9-bx4l2"
Sep 14 10:00:05.659492 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:05.659371    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90443638-9dd8-4c4d-812f-146cba869af7-tls-certs\") pod \"test-model-0e60f3-kserve-84c9545bb9-bx4l2\" (UID: \"90443638-9dd8-4c4d-812f-146cba869af7\") " pod="ai-tenant-e2e-inf-a-0e60f3/test-model-0e60f3-kserve-84c9545bb9-bx4l2"
Sep 14 10:00:05.664907 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:05.664876    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qvrj\" (UniqueName: \"kubernetes.io/projected/90443638-9dd8-4c4d-812f-146cba869af7-kube-api-access-7qvrj\") pod \"test-model-0e60f3-kserve-84c9545bb9-bx4l2\" (UID: \"90443638-9dd8-4c4d-812f-146cba869af7\") " pod="ai-tenant-e2e-inf-a-0e60f3/test-model-0e60f3-kserve-84c9545bb9-bx4l2"
Sep 14 10:00:05.777420 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:05.777379    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0e60f3/test-model-0e60f3-kserve-84c9545bb9-bx4l2"
Sep 14 10:00:06.025220 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:06.025182    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:00:06.064240 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:06.064197    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-0e60f3/test-model-0e60f3-kserve-84c9545bb9-bx4l2"]
Sep 14 10:00:06.072638 ip-10-0-132-4 kubenswrapper[2563]: W0914 10:00:06.072602    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod90443638_9dd8_4c4d_812f_146cba869af7.slice/crio-77f775bb3d18b233a57588c976ad37803f250e27225acddf0a4c929a0c1c66ae WatchSource:0}: Error finding container 77f775bb3d18b233a57588c976ad37803f250e27225acddf0a4c929a0c1c66ae: Status 404 returned error can't find the container with id 77f775bb3d18b233a57588c976ad37803f250e27225acddf0a4c929a0c1c66ae
Sep 14 10:00:06.144634 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:06.144594    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0e60f3/test-model-0e60f3-kserve-84c9545bb9-bx4l2" event={"ID":"90443638-9dd8-4c4d-812f-146cba869af7","Type":"ContainerStarted","Data":"b658770c20270cbbb20fa5b41b339d2e5764b3719099f9d47c8a8061966c23cf"}
Sep 14 10:00:06.144813 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:06.144647    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0e60f3/test-model-0e60f3-kserve-84c9545bb9-bx4l2" event={"ID":"90443638-9dd8-4c4d-812f-146cba869af7","Type":"ContainerStarted","Data":"77f775bb3d18b233a57588c976ad37803f250e27225acddf0a4c929a0c1c66ae"}
Sep 14 10:00:06.164476 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:06.164417    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-0e60f3/test-model-0e60f3-kserve-84c9545bb9-bx4l2" podStartSLOduration=1.164397642 podStartE2EDuration="1.164397642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:00:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:00:06.162771559 +0000 UTC m=+995.202446179" watchObservedRunningTime="2026-09-14 10:00:06.164397642 +0000 UTC m=+995.204072262"
Sep 14 10:00:06.338648 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:06.338619    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm"
Sep 14 10:00:06.364834 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:06.364798    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/309ac5a4-3502-4351-830d-e6cf50ab73be-istio-data\") pod \"309ac5a4-3502-4351-830d-e6cf50ab73be\" (UID: \"309ac5a4-3502-4351-830d-e6cf50ab73be\") "
Sep 14 10:00:06.365061 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:06.364872    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/309ac5a4-3502-4351-830d-e6cf50ab73be-istiod-ca-cert\") pod \"309ac5a4-3502-4351-830d-e6cf50ab73be\" (UID: \"309ac5a4-3502-4351-830d-e6cf50ab73be\") "
Sep 14 10:00:06.365061 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:06.364901    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/309ac5a4-3502-4351-830d-e6cf50ab73be-workload-socket\") pod \"309ac5a4-3502-4351-830d-e6cf50ab73be\" (UID: \"309ac5a4-3502-4351-830d-e6cf50ab73be\") "
Sep 14 10:00:06.365061 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:06.365020    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/309ac5a4-3502-4351-830d-e6cf50ab73be-workload-certs\") pod \"309ac5a4-3502-4351-830d-e6cf50ab73be\" (UID: \"309ac5a4-3502-4351-830d-e6cf50ab73be\") "
Sep 14 10:00:06.365061 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:06.365058    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/309ac5a4-3502-4351-830d-e6cf50ab73be-istio-token\") pod \"309ac5a4-3502-4351-830d-e6cf50ab73be\" (UID: \"309ac5a4-3502-4351-830d-e6cf50ab73be\") "
Sep 14 10:00:06.365310 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:06.365101    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/309ac5a4-3502-4351-830d-e6cf50ab73be-credential-socket\") pod \"309ac5a4-3502-4351-830d-e6cf50ab73be\" (UID: \"309ac5a4-3502-4351-830d-e6cf50ab73be\") "
Sep 14 10:00:06.365310 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:06.365109    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/309ac5a4-3502-4351-830d-e6cf50ab73be-istio-data" (OuterVolumeSpecName: "istio-data") pod "309ac5a4-3502-4351-830d-e6cf50ab73be" (UID: "309ac5a4-3502-4351-830d-e6cf50ab73be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:00:06.365310 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:06.365141    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-67tqf\" (UniqueName: \"kubernetes.io/projected/309ac5a4-3502-4351-830d-e6cf50ab73be-kube-api-access-67tqf\") pod \"309ac5a4-3502-4351-830d-e6cf50ab73be\" (UID: \"309ac5a4-3502-4351-830d-e6cf50ab73be\") "
Sep 14 10:00:06.365310 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:06.365219    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/309ac5a4-3502-4351-830d-e6cf50ab73be-istio-envoy\") pod \"309ac5a4-3502-4351-830d-e6cf50ab73be\" (UID: \"309ac5a4-3502-4351-830d-e6cf50ab73be\") "
Sep 14 10:00:06.365310 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:06.365249    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/309ac5a4-3502-4351-830d-e6cf50ab73be-istio-podinfo\") pod \"309ac5a4-3502-4351-830d-e6cf50ab73be\" (UID: \"309ac5a4-3502-4351-830d-e6cf50ab73be\") "
Sep 14 10:00:06.365310 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:06.365262    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/309ac5a4-3502-4351-830d-e6cf50ab73be-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "309ac5a4-3502-4351-830d-e6cf50ab73be" (UID: "309ac5a4-3502-4351-830d-e6cf50ab73be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:00:06.365605 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:06.365422    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/309ac5a4-3502-4351-830d-e6cf50ab73be-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "309ac5a4-3502-4351-830d-e6cf50ab73be" (UID: "309ac5a4-3502-4351-830d-e6cf50ab73be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:00:06.365689 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:06.365654    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/309ac5a4-3502-4351-830d-e6cf50ab73be-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:00:06.365689 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:06.365683    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/309ac5a4-3502-4351-830d-e6cf50ab73be-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:00:06.365792 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:06.365699    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/309ac5a4-3502-4351-830d-e6cf50ab73be-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:00:06.365792 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:06.365706    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/309ac5a4-3502-4351-830d-e6cf50ab73be-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "309ac5a4-3502-4351-830d-e6cf50ab73be" (UID: "309ac5a4-3502-4351-830d-e6cf50ab73be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:00:06.366306 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:06.366276    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/309ac5a4-3502-4351-830d-e6cf50ab73be-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "309ac5a4-3502-4351-830d-e6cf50ab73be" (UID: "309ac5a4-3502-4351-830d-e6cf50ab73be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:00:06.368350 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:06.368317    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/309ac5a4-3502-4351-830d-e6cf50ab73be-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "309ac5a4-3502-4351-830d-e6cf50ab73be" (UID: "309ac5a4-3502-4351-830d-e6cf50ab73be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:00:06.368492 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:06.368322    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/309ac5a4-3502-4351-830d-e6cf50ab73be-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "309ac5a4-3502-4351-830d-e6cf50ab73be" (UID: "309ac5a4-3502-4351-830d-e6cf50ab73be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 10:00:06.368554 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:06.368503    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/309ac5a4-3502-4351-830d-e6cf50ab73be-istio-token" (OuterVolumeSpecName: "istio-token") pod "309ac5a4-3502-4351-830d-e6cf50ab73be" (UID: "309ac5a4-3502-4351-830d-e6cf50ab73be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:00:06.368634 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:06.368611    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/309ac5a4-3502-4351-830d-e6cf50ab73be-kube-api-access-67tqf" (OuterVolumeSpecName: "kube-api-access-67tqf") pod "309ac5a4-3502-4351-830d-e6cf50ab73be" (UID: "309ac5a4-3502-4351-830d-e6cf50ab73be"). InnerVolumeSpecName "kube-api-access-67tqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:00:06.466724 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:06.466683    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/309ac5a4-3502-4351-830d-e6cf50ab73be-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:00:06.466724 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:06.466718    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/309ac5a4-3502-4351-830d-e6cf50ab73be-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:00:06.466724 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:06.466726    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/309ac5a4-3502-4351-830d-e6cf50ab73be-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:00:06.466724 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:06.466734    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/309ac5a4-3502-4351-830d-e6cf50ab73be-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:00:06.467050 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:06.466742    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/309ac5a4-3502-4351-830d-e6cf50ab73be-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:00:06.467050 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:06.466751    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67tqf\" (UniqueName: \"kubernetes.io/projected/309ac5a4-3502-4351-830d-e6cf50ab73be-kube-api-access-67tqf\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:00:06.777920 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:06.777883    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:00:07.028930 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:07.028894    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:00:07.154036 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:07.153985    2563 generic.go:356] "Generic (PLEG): container finished" podID="309ac5a4-3502-4351-830d-e6cf50ab73be" containerID="560183e951ece993d34ee8f1436b82ad8f228500716d64a592daea81e3df3724" exitCode=0
Sep 14 10:00:07.154217 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:07.154059    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm"
Sep 14 10:00:07.154217 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:07.154073    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm" event={"ID":"309ac5a4-3502-4351-830d-e6cf50ab73be","Type":"ContainerDied","Data":"560183e951ece993d34ee8f1436b82ad8f228500716d64a592daea81e3df3724"}
Sep 14 10:00:07.154217 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:07.154144    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm" event={"ID":"309ac5a4-3502-4351-830d-e6cf50ab73be","Type":"ContainerDied","Data":"7a7f09c6bc969f5eac3b1d8f51b171cd8233c9de2423ab042dfe33c9a1674ae1"}
Sep 14 10:00:07.154217 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:07.154176    2563 scope.go:117] "RemoveContainer" containerID="560183e951ece993d34ee8f1436b82ad8f228500716d64a592daea81e3df3724"
Sep 14 10:00:07.165494 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:07.165470    2563 scope.go:117] "RemoveContainer" containerID="f9fabeae6867291a9e9ca5da8d5aef251af47ecb90ac1fe8f7c7de5619ff90a3"
Sep 14 10:00:07.177341 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:07.177315    2563 scope.go:117] "RemoveContainer" containerID="560183e951ece993d34ee8f1436b82ad8f228500716d64a592daea81e3df3724"
Sep 14 10:00:07.177706 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:07.177683    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm"]
Sep 14 10:00:07.177780 ip-10-0-132-4 kubenswrapper[2563]: E0914 10:00:07.177702    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"560183e951ece993d34ee8f1436b82ad8f228500716d64a592daea81e3df3724\": container with ID starting with 560183e951ece993d34ee8f1436b82ad8f228500716d64a592daea81e3df3724 not found: ID does not exist" containerID="560183e951ece993d34ee8f1436b82ad8f228500716d64a592daea81e3df3724"
Sep 14 10:00:07.177780 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:07.177732    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"560183e951ece993d34ee8f1436b82ad8f228500716d64a592daea81e3df3724"} err="failed to get container status \"560183e951ece993d34ee8f1436b82ad8f228500716d64a592daea81e3df3724\": rpc error: code = NotFound desc = could not find container \"560183e951ece993d34ee8f1436b82ad8f228500716d64a592daea81e3df3724\": container with ID starting with 560183e951ece993d34ee8f1436b82ad8f228500716d64a592daea81e3df3724 not found: ID does not exist"
Sep 14 10:00:07.177780 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:07.177752    2563 scope.go:117] "RemoveContainer" containerID="f9fabeae6867291a9e9ca5da8d5aef251af47ecb90ac1fe8f7c7de5619ff90a3"
Sep 14 10:00:07.178128 ip-10-0-132-4 kubenswrapper[2563]: E0914 10:00:07.178102    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9fabeae6867291a9e9ca5da8d5aef251af47ecb90ac1fe8f7c7de5619ff90a3\": container with ID starting with f9fabeae6867291a9e9ca5da8d5aef251af47ecb90ac1fe8f7c7de5619ff90a3 not found: ID does not exist" containerID="f9fabeae6867291a9e9ca5da8d5aef251af47ecb90ac1fe8f7c7de5619ff90a3"
Sep 14 10:00:07.178189 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:07.178140    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9fabeae6867291a9e9ca5da8d5aef251af47ecb90ac1fe8f7c7de5619ff90a3"} err="failed to get container status \"f9fabeae6867291a9e9ca5da8d5aef251af47ecb90ac1fe8f7c7de5619ff90a3\": rpc error: code = NotFound desc = could not find container \"f9fabeae6867291a9e9ca5da8d5aef251af47ecb90ac1fe8f7c7de5619ff90a3\": container with ID starting with f9fabeae6867291a9e9ca5da8d5aef251af47ecb90ac1fe8f7c7de5619ff90a3 not found: ID does not exist"
Sep 14 10:00:07.182179 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:07.182137    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-80fa7537-openshift-default-56f89f8d64-nkhmm"]
Sep 14 10:00:07.483932 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:07.483836    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="309ac5a4-3502-4351-830d-e6cf50ab73be" path="/var/lib/kubelet/pods/309ac5a4-3502-4351-830d-e6cf50ab73be/volumes"
Sep 14 10:00:11.023429 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:11.023383    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:00:11.451636 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:11.451596    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:00:11.573402 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:11.573367    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:00:12.139903 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:12.139867    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:00:12.234440 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:12.234398    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-8c98aa-29823000-6dbrt"]
Sep 14 10:00:12.239987 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:12.239946    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-8c98aa-29823000-6dbrt"]
Sep 14 10:00:12.241622 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:12.241592    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:00:12.317243 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:12.317211    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:00:12.468313 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:12.468213    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:00:12.544297 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:12.544256    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:00:12.861882 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:12.861844    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:00:12.970806 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:12.970771    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:00:13.482967 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:13.482931    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="116b063c-bcb9-4a35-91c6-4c894aec8d4a" path="/var/lib/kubelet/pods/116b063c-bcb9-4a35-91c6-4c894aec8d4a/volumes"
Sep 14 10:00:13.860533 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:13.860501    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:00:14.816827 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:14.816791    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:00:15.703671 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:15.703634    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:00:15.777827 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:15.777787    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-0e60f3/test-model-0e60f3-kserve-84c9545bb9-bx4l2"
Sep 14 10:00:15.778064 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:15.777852    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-0e60f3/test-model-0e60f3-kserve-84c9545bb9-bx4l2"
Sep 14 10:00:15.791106 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:15.791078    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-0e60f3/test-model-0e60f3-kserve-84c9545bb9-bx4l2"
Sep 14 10:00:16.225924 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:16.225885    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-0e60f3/test-model-0e60f3-kserve-84c9545bb9-bx4l2"
Sep 14 10:00:16.618202 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:16.618161    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:00:17.944355 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:17.944319    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:00:21.375907 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:21.375865    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:00:21.870727 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:21.870693    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:00:28.487958 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:28.487920    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:00:37.115670 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:37.115630    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:00:41.398944 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:41.398911    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:00:43.541082 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:43.541043    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:00:43.678395 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:43.678359    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:00:48.161884 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:48.161842    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:00:57.117151 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:57.117111    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn"]
Sep 14 10:00:57.117652 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:57.117423    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn" podUID="eae8537b-56e1-4d2a-a43b-de5bb82f78b8" containerName="istio-proxy" containerID="cri-o://bd54bf9c93704b7fcef1b04e0b006646544ad125d4a515355c04bf6db9cb205e" gracePeriod=30
Sep 14 10:00:57.120259 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:57.120211    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:00:57.261634 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:57.261595    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:00:57.582690 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:57.582655    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:00:57.891060 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:57.890935    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48"]
Sep 14 10:00:57.891372 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:57.891357    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="309ac5a4-3502-4351-830d-e6cf50ab73be" containerName="istio-proxy"
Sep 14 10:00:57.891422 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:57.891373    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="309ac5a4-3502-4351-830d-e6cf50ab73be" containerName="istio-proxy"
Sep 14 10:00:57.891459 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:57.891451    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="309ac5a4-3502-4351-830d-e6cf50ab73be" containerName="istio-proxy"
Sep 14 10:00:57.895947 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:57.895923    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48"
Sep 14 10:00:57.898502 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:57.898476    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-31f3b6-openshift-default-dockercfg-xw46c\""
Sep 14 10:00:57.908182 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:57.908143    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48"]
Sep 14 10:00:57.925176 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:57.925133    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:00:58.014685 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:58.014634    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4jjr\" (UniqueName: \"kubernetes.io/projected/6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d-kube-api-access-j4jjr\") pod \"e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48\" (UID: \"6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d\") " pod="openshift-ingress/e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48"
Sep 14 10:00:58.014907 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:58.014697    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d-workload-certs\") pod \"e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48\" (UID: \"6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d\") " pod="openshift-ingress/e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48"
Sep 14 10:00:58.014907 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:58.014760    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d-istio-token\") pod \"e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48\" (UID: \"6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d\") " pod="openshift-ingress/e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48"
Sep 14 10:00:58.014907 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:58.014798    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d-workload-socket\") pod \"e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48\" (UID: \"6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d\") " pod="openshift-ingress/e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48"
Sep 14 10:00:58.014907 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:58.014833    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d-istio-podinfo\") pod \"e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48\" (UID: \"6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d\") " pod="openshift-ingress/e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48"
Sep 14 10:00:58.014907 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:58.014868    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d-istiod-ca-cert\") pod \"e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48\" (UID: \"6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d\") " pod="openshift-ingress/e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48"
Sep 14 10:00:58.014907 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:58.014903    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d-credential-socket\") pod \"e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48\" (UID: \"6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d\") " pod="openshift-ingress/e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48"
Sep 14 10:00:58.015241 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:58.014974    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d-istio-data\") pod \"e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48\" (UID: \"6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d\") " pod="openshift-ingress/e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48"
Sep 14 10:00:58.015241 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:58.015051    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d-istio-envoy\") pod \"e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48\" (UID: \"6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d\") " pod="openshift-ingress/e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48"
Sep 14 10:00:58.116362 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:58.116318    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d-istio-envoy\") pod \"e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48\" (UID: \"6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d\") " pod="openshift-ingress/e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48"
Sep 14 10:00:58.116583 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:58.116393    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4jjr\" (UniqueName: \"kubernetes.io/projected/6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d-kube-api-access-j4jjr\") pod \"e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48\" (UID: \"6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d\") " pod="openshift-ingress/e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48"
Sep 14 10:00:58.116583 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:58.116423    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d-workload-certs\") pod \"e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48\" (UID: \"6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d\") " pod="openshift-ingress/e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48"
Sep 14 10:00:58.116583 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:58.116479    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d-istio-token\") pod \"e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48\" (UID: \"6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d\") " pod="openshift-ingress/e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48"
Sep 14 10:00:58.116583 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:58.116515    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d-workload-socket\") pod \"e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48\" (UID: \"6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d\") " pod="openshift-ingress/e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48"
Sep 14 10:00:58.116583 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:58.116544    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d-istio-podinfo\") pod \"e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48\" (UID: \"6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d\") " pod="openshift-ingress/e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48"
Sep 14 10:00:58.116583 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:58.116578    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d-istiod-ca-cert\") pod \"e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48\" (UID: \"6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d\") " pod="openshift-ingress/e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48"
Sep 14 10:00:58.116913 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:58.116641    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d-credential-socket\") pod \"e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48\" (UID: \"6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d\") " pod="openshift-ingress/e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48"
Sep 14 10:00:58.116913 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:58.116820    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d-istio-data\") pod \"e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48\" (UID: \"6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d\") " pod="openshift-ingress/e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48"
Sep 14 10:00:58.116913 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:58.116896    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d-workload-certs\") pod \"e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48\" (UID: \"6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d\") " pod="openshift-ingress/e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48"
Sep 14 10:00:58.117117 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:58.116920    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d-workload-socket\") pod \"e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48\" (UID: \"6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d\") " pod="openshift-ingress/e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48"
Sep 14 10:00:58.117252 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:58.117225    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d-credential-socket\") pod \"e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48\" (UID: \"6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d\") " pod="openshift-ingress/e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48"
Sep 14 10:00:58.117546 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:58.117237    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d-istio-data\") pod \"e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48\" (UID: \"6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d\") " pod="openshift-ingress/e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48"
Sep 14 10:00:58.117546 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:58.117320    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d-istiod-ca-cert\") pod \"e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48\" (UID: \"6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d\") " pod="openshift-ingress/e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48"
Sep 14 10:00:58.118870 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:58.118848    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d-istio-envoy\") pod \"e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48\" (UID: \"6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d\") " pod="openshift-ingress/e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48"
Sep 14 10:00:58.119130 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:58.119109    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d-istio-podinfo\") pod \"e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48\" (UID: \"6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d\") " pod="openshift-ingress/e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48"
Sep 14 10:00:58.125218 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:58.125188    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d-istio-token\") pod \"e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48\" (UID: \"6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d\") " pod="openshift-ingress/e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48"
Sep 14 10:00:58.125364 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:58.125321    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4jjr\" (UniqueName: \"kubernetes.io/projected/6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d-kube-api-access-j4jjr\") pod \"e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48\" (UID: \"6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d\") " pod="openshift-ingress/e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48"
Sep 14 10:00:58.157100 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:58.156966    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:00:58.209133 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:58.209082    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48"
Sep 14 10:00:58.503078 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:58.503045    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48"]
Sep 14 10:00:58.505433 ip-10-0-132-4 kubenswrapper[2563]: W0914 10:00:58.505385    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ebfd43a_3338_44c7_8c43_5f42c3a5ad1d.slice/crio-171abaaaca437829182c694ed964ecb747c77f50d098daba1f3f96e50fa48493 WatchSource:0}: Error finding container 171abaaaca437829182c694ed964ecb747c77f50d098daba1f3f96e50fa48493: Status 404 returned error can't find the container with id 171abaaaca437829182c694ed964ecb747c77f50d098daba1f3f96e50fa48493
Sep 14 10:00:58.507881 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:58.507839    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 10:00:58.508028 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:58.507934    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 10:00:58.508028 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:58.507980    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 10:00:58.787409 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:58.787323    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:00:58.940726 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:58.940690    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:00:59.491799 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:59.491758    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48" event={"ID":"6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d","Type":"ContainerStarted","Data":"a73865c4e4cf5831364578f49708bffeb619db64eecfcae81eb51b77470e959c"}
Sep 14 10:00:59.492444 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:59.491826    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48" event={"ID":"6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d","Type":"ContainerStarted","Data":"171abaaaca437829182c694ed964ecb747c77f50d098daba1f3f96e50fa48493"}
Sep 14 10:00:59.517508 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:00:59.517463    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:01:00.209901 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:00.209862    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48"
Sep 14 10:01:00.215289 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:00.215260    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48"
Sep 14 10:01:00.242284 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:00.242219    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48" podStartSLOduration=3.242198192 podStartE2EDuration="3.242198192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:00:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:00:59.543644924 +0000 UTC m=+1048.583319598" watchObservedRunningTime="2026-09-14 10:01:00.242198192 +0000 UTC m=+1049.281872880"
Sep 14 10:01:00.498849 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:00.498813    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48"
Sep 14 10:01:00.499875 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:00.499851    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48"
Sep 14 10:01:00.921948 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:00.921857    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:01:01.508780 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:01.508704    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:01:01.610761 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:01.610706    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:01:01.930851 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:01.930740    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:01:02.376304 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:02.376267    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn"
Sep 14 10:01:02.513136 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:02.513101    2563 generic.go:356] "Generic (PLEG): container finished" podID="eae8537b-56e1-4d2a-a43b-de5bb82f78b8" containerID="bd54bf9c93704b7fcef1b04e0b006646544ad125d4a515355c04bf6db9cb205e" exitCode=0
Sep 14 10:01:02.513604 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:02.513168    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn"
Sep 14 10:01:02.513604 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:02.513184    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn" event={"ID":"eae8537b-56e1-4d2a-a43b-de5bb82f78b8","Type":"ContainerDied","Data":"bd54bf9c93704b7fcef1b04e0b006646544ad125d4a515355c04bf6db9cb205e"}
Sep 14 10:01:02.513604 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:02.513235    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn" event={"ID":"eae8537b-56e1-4d2a-a43b-de5bb82f78b8","Type":"ContainerDied","Data":"4c123d948cfd934575fca0e26e66bf66960deb4fa81e8f7aab09506a96da4df4"}
Sep 14 10:01:02.513604 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:02.513261    2563 scope.go:117] "RemoveContainer" containerID="bd54bf9c93704b7fcef1b04e0b006646544ad125d4a515355c04bf6db9cb205e"
Sep 14 10:01:02.523564 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:02.523541    2563 scope.go:117] "RemoveContainer" containerID="8afcbeade6f146478a38713532fae180f6a9baa54dfaa533506ce9ad77f83340"
Sep 14 10:01:02.533738 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:02.533715    2563 scope.go:117] "RemoveContainer" containerID="bd54bf9c93704b7fcef1b04e0b006646544ad125d4a515355c04bf6db9cb205e"
Sep 14 10:01:02.534143 ip-10-0-132-4 kubenswrapper[2563]: E0914 10:01:02.534119    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd54bf9c93704b7fcef1b04e0b006646544ad125d4a515355c04bf6db9cb205e\": container with ID starting with bd54bf9c93704b7fcef1b04e0b006646544ad125d4a515355c04bf6db9cb205e not found: ID does not exist" containerID="bd54bf9c93704b7fcef1b04e0b006646544ad125d4a515355c04bf6db9cb205e"
Sep 14 10:01:02.534214 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:02.534157    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd54bf9c93704b7fcef1b04e0b006646544ad125d4a515355c04bf6db9cb205e"} err="failed to get container status \"bd54bf9c93704b7fcef1b04e0b006646544ad125d4a515355c04bf6db9cb205e\": rpc error: code = NotFound desc = could not find container \"bd54bf9c93704b7fcef1b04e0b006646544ad125d4a515355c04bf6db9cb205e\": container with ID starting with bd54bf9c93704b7fcef1b04e0b006646544ad125d4a515355c04bf6db9cb205e not found: ID does not exist"
Sep 14 10:01:02.534214 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:02.534177    2563 scope.go:117] "RemoveContainer" containerID="8afcbeade6f146478a38713532fae180f6a9baa54dfaa533506ce9ad77f83340"
Sep 14 10:01:02.534487 ip-10-0-132-4 kubenswrapper[2563]: E0914 10:01:02.534469    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8afcbeade6f146478a38713532fae180f6a9baa54dfaa533506ce9ad77f83340\": container with ID starting with 8afcbeade6f146478a38713532fae180f6a9baa54dfaa533506ce9ad77f83340 not found: ID does not exist" containerID="8afcbeade6f146478a38713532fae180f6a9baa54dfaa533506ce9ad77f83340"
Sep 14 10:01:02.534525 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:02.534495    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8afcbeade6f146478a38713532fae180f6a9baa54dfaa533506ce9ad77f83340"} err="failed to get container status \"8afcbeade6f146478a38713532fae180f6a9baa54dfaa533506ce9ad77f83340\": rpc error: code = NotFound desc = could not find container \"8afcbeade6f146478a38713532fae180f6a9baa54dfaa533506ce9ad77f83340\": container with ID starting with 8afcbeade6f146478a38713532fae180f6a9baa54dfaa533506ce9ad77f83340 not found: ID does not exist"
Sep 14 10:01:02.564869 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:02.564830    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eae8537b-56e1-4d2a-a43b-de5bb82f78b8-credential-socket\") pod \"eae8537b-56e1-4d2a-a43b-de5bb82f78b8\" (UID: \"eae8537b-56e1-4d2a-a43b-de5bb82f78b8\") "
Sep 14 10:01:02.565031 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:02.564882    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eae8537b-56e1-4d2a-a43b-de5bb82f78b8-workload-certs\") pod \"eae8537b-56e1-4d2a-a43b-de5bb82f78b8\" (UID: \"eae8537b-56e1-4d2a-a43b-de5bb82f78b8\") "
Sep 14 10:01:02.565031 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:02.564915    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eae8537b-56e1-4d2a-a43b-de5bb82f78b8-istio-podinfo\") pod \"eae8537b-56e1-4d2a-a43b-de5bb82f78b8\" (UID: \"eae8537b-56e1-4d2a-a43b-de5bb82f78b8\") "
Sep 14 10:01:02.565031 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:02.564946    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eae8537b-56e1-4d2a-a43b-de5bb82f78b8-istiod-ca-cert\") pod \"eae8537b-56e1-4d2a-a43b-de5bb82f78b8\" (UID: \"eae8537b-56e1-4d2a-a43b-de5bb82f78b8\") "
Sep 14 10:01:02.565299 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:02.565265    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eae8537b-56e1-4d2a-a43b-de5bb82f78b8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "eae8537b-56e1-4d2a-a43b-de5bb82f78b8" (UID: "eae8537b-56e1-4d2a-a43b-de5bb82f78b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:01:02.565299 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:02.565290    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eae8537b-56e1-4d2a-a43b-de5bb82f78b8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "eae8537b-56e1-4d2a-a43b-de5bb82f78b8" (UID: "eae8537b-56e1-4d2a-a43b-de5bb82f78b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:01:02.565471 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:02.565332    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eae8537b-56e1-4d2a-a43b-de5bb82f78b8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "eae8537b-56e1-4d2a-a43b-de5bb82f78b8" (UID: "eae8537b-56e1-4d2a-a43b-de5bb82f78b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:01:02.565471 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:02.565411    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eae8537b-56e1-4d2a-a43b-de5bb82f78b8-istio-token\") pod \"eae8537b-56e1-4d2a-a43b-de5bb82f78b8\" (UID: \"eae8537b-56e1-4d2a-a43b-de5bb82f78b8\") "
Sep 14 10:01:02.565471 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:02.565469    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eae8537b-56e1-4d2a-a43b-de5bb82f78b8-istio-envoy\") pod \"eae8537b-56e1-4d2a-a43b-de5bb82f78b8\" (UID: \"eae8537b-56e1-4d2a-a43b-de5bb82f78b8\") "
Sep 14 10:01:02.565601 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:02.565501    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eae8537b-56e1-4d2a-a43b-de5bb82f78b8-workload-socket\") pod \"eae8537b-56e1-4d2a-a43b-de5bb82f78b8\" (UID: \"eae8537b-56e1-4d2a-a43b-de5bb82f78b8\") "
Sep 14 10:01:02.565601 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:02.565534    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g565h\" (UniqueName: \"kubernetes.io/projected/eae8537b-56e1-4d2a-a43b-de5bb82f78b8-kube-api-access-g565h\") pod \"eae8537b-56e1-4d2a-a43b-de5bb82f78b8\" (UID: \"eae8537b-56e1-4d2a-a43b-de5bb82f78b8\") "
Sep 14 10:01:02.565601 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:02.565595    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eae8537b-56e1-4d2a-a43b-de5bb82f78b8-istio-data\") pod \"eae8537b-56e1-4d2a-a43b-de5bb82f78b8\" (UID: \"eae8537b-56e1-4d2a-a43b-de5bb82f78b8\") "
Sep 14 10:01:02.566107 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:02.565802    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eae8537b-56e1-4d2a-a43b-de5bb82f78b8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "eae8537b-56e1-4d2a-a43b-de5bb82f78b8" (UID: "eae8537b-56e1-4d2a-a43b-de5bb82f78b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:01:02.566807 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:02.566357    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eae8537b-56e1-4d2a-a43b-de5bb82f78b8-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:01:02.566807 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:02.566396    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eae8537b-56e1-4d2a-a43b-de5bb82f78b8-istio-data" (OuterVolumeSpecName: "istio-data") pod "eae8537b-56e1-4d2a-a43b-de5bb82f78b8" (UID: "eae8537b-56e1-4d2a-a43b-de5bb82f78b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:01:02.566807 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:02.566458    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eae8537b-56e1-4d2a-a43b-de5bb82f78b8-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:01:02.566807 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:02.566479    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eae8537b-56e1-4d2a-a43b-de5bb82f78b8-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:01:02.566807 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:02.566497    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eae8537b-56e1-4d2a-a43b-de5bb82f78b8-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:01:02.567561 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:02.567538    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eae8537b-56e1-4d2a-a43b-de5bb82f78b8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "eae8537b-56e1-4d2a-a43b-de5bb82f78b8" (UID: "eae8537b-56e1-4d2a-a43b-de5bb82f78b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 10:01:02.567884 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:02.567842    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eae8537b-56e1-4d2a-a43b-de5bb82f78b8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "eae8537b-56e1-4d2a-a43b-de5bb82f78b8" (UID: "eae8537b-56e1-4d2a-a43b-de5bb82f78b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:01:02.567884 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:02.567862    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eae8537b-56e1-4d2a-a43b-de5bb82f78b8-kube-api-access-g565h" (OuterVolumeSpecName: "kube-api-access-g565h") pod "eae8537b-56e1-4d2a-a43b-de5bb82f78b8" (UID: "eae8537b-56e1-4d2a-a43b-de5bb82f78b8"). InnerVolumeSpecName "kube-api-access-g565h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:01:02.568093 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:02.567932    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eae8537b-56e1-4d2a-a43b-de5bb82f78b8-istio-token" (OuterVolumeSpecName: "istio-token") pod "eae8537b-56e1-4d2a-a43b-de5bb82f78b8" (UID: "eae8537b-56e1-4d2a-a43b-de5bb82f78b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:01:02.667399 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:02.667353    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g565h\" (UniqueName: \"kubernetes.io/projected/eae8537b-56e1-4d2a-a43b-de5bb82f78b8-kube-api-access-g565h\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:01:02.667399 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:02.667395    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eae8537b-56e1-4d2a-a43b-de5bb82f78b8-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:01:02.667399 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:02.667406    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eae8537b-56e1-4d2a-a43b-de5bb82f78b8-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:01:02.667399 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:02.667414    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eae8537b-56e1-4d2a-a43b-de5bb82f78b8-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:01:02.667684 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:02.667422    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eae8537b-56e1-4d2a-a43b-de5bb82f78b8-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:01:02.851023 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:02.845724    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn"]
Sep 14 10:01:02.855461 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:02.855423    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-8c98aa-openshift-default-56775dc6f8-p45mn"]
Sep 14 10:01:03.217131 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:03.217034    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:01:03.482431 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:03.482344    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eae8537b-56e1-4d2a-a43b-de5bb82f78b8" path="/var/lib/kubelet/pods/eae8537b-56e1-4d2a-a43b-de5bb82f78b8/volumes"
Sep 14 10:01:04.145945 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:04.145904    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:01:05.522640 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:05.522607    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:01:06.526230 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:06.526196    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:01:07.228719 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:07.228661    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:01:07.330194 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:07.330155    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:01:17.581364 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:17.581325    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:01:18.441882 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:18.441842    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:01:24.831884 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:24.831841    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:01:24.938817 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:24.938776    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:01:25.058153 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:25.058114    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:01:25.157425 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:25.157333    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:01:25.980602 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:25.980556    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:01:26.445118 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:26.445082    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:01:27.171822 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:27.171725    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-24873d/test-model-24873d-kserve-dff86779c-29lrm"]
Sep 14 10:01:27.172314 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:27.172165    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eae8537b-56e1-4d2a-a43b-de5bb82f78b8" containerName="istio-proxy"
Sep 14 10:01:27.172314 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:27.172179    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="eae8537b-56e1-4d2a-a43b-de5bb82f78b8" containerName="istio-proxy"
Sep 14 10:01:27.172314 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:27.172203    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="309ac5a4-3502-4351-830d-e6cf50ab73be" containerName="istio-proxy"
Sep 14 10:01:27.172314 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:27.172209    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="309ac5a4-3502-4351-830d-e6cf50ab73be" containerName="istio-proxy"
Sep 14 10:01:27.172314 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:27.172293    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="eae8537b-56e1-4d2a-a43b-de5bb82f78b8" containerName="istio-proxy"
Sep 14 10:01:27.172314 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:27.172301    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="309ac5a4-3502-4351-830d-e6cf50ab73be" containerName="istio-proxy"
Sep 14 10:01:27.172314 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:27.172310    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="eae8537b-56e1-4d2a-a43b-de5bb82f78b8" containerName="istio-proxy"
Sep 14 10:01:27.174629 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:27.174606    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-24873d/test-model-24873d-kserve-dff86779c-29lrm"
Sep 14 10:01:27.179373 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:27.179340    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-24873d\"/\"openshift-service-ca.crt\""
Sep 14 10:01:27.179845 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:27.179826    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-24873d\"/\"kube-root-ca.crt\""
Sep 14 10:01:27.180039 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:27.180021    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-24873d\"/\"test-model-24873d-kserve-self-signed-certs\""
Sep 14 10:01:27.180139 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:27.180059    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-24873d\"/\"default-dockercfg-np8s9\""
Sep 14 10:01:27.196203 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:27.196164    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-24873d/test-model-24873d-kserve-dff86779c-29lrm"]
Sep 14 10:01:27.325600 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:27.325560    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d55a291-49c9-4c1e-8ae1-7cf6632d4395-model-cache\") pod \"test-model-24873d-kserve-dff86779c-29lrm\" (UID: \"0d55a291-49c9-4c1e-8ae1-7cf6632d4395\") " pod="ai-tenant-e2e-inf-b-24873d/test-model-24873d-kserve-dff86779c-29lrm"
Sep 14 10:01:27.325600 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:27.325606    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d55a291-49c9-4c1e-8ae1-7cf6632d4395-dshm\") pod \"test-model-24873d-kserve-dff86779c-29lrm\" (UID: \"0d55a291-49c9-4c1e-8ae1-7cf6632d4395\") " pod="ai-tenant-e2e-inf-b-24873d/test-model-24873d-kserve-dff86779c-29lrm"
Sep 14 10:01:27.325834 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:27.325632    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d55a291-49c9-4c1e-8ae1-7cf6632d4395-tmp-dir\") pod \"test-model-24873d-kserve-dff86779c-29lrm\" (UID: \"0d55a291-49c9-4c1e-8ae1-7cf6632d4395\") " pod="ai-tenant-e2e-inf-b-24873d/test-model-24873d-kserve-dff86779c-29lrm"
Sep 14 10:01:27.325834 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:27.325717    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d55a291-49c9-4c1e-8ae1-7cf6632d4395-tls-certs\") pod \"test-model-24873d-kserve-dff86779c-29lrm\" (UID: \"0d55a291-49c9-4c1e-8ae1-7cf6632d4395\") " pod="ai-tenant-e2e-inf-b-24873d/test-model-24873d-kserve-dff86779c-29lrm"
Sep 14 10:01:27.325834 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:27.325765    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d55a291-49c9-4c1e-8ae1-7cf6632d4395-home\") pod \"test-model-24873d-kserve-dff86779c-29lrm\" (UID: \"0d55a291-49c9-4c1e-8ae1-7cf6632d4395\") " pod="ai-tenant-e2e-inf-b-24873d/test-model-24873d-kserve-dff86779c-29lrm"
Sep 14 10:01:27.325834 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:27.325804    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx88v\" (UniqueName: \"kubernetes.io/projected/0d55a291-49c9-4c1e-8ae1-7cf6632d4395-kube-api-access-gx88v\") pod \"test-model-24873d-kserve-dff86779c-29lrm\" (UID: \"0d55a291-49c9-4c1e-8ae1-7cf6632d4395\") " pod="ai-tenant-e2e-inf-b-24873d/test-model-24873d-kserve-dff86779c-29lrm"
Sep 14 10:01:27.346521 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:27.346484    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:01:27.426772 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:27.426672    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d55a291-49c9-4c1e-8ae1-7cf6632d4395-model-cache\") pod \"test-model-24873d-kserve-dff86779c-29lrm\" (UID: \"0d55a291-49c9-4c1e-8ae1-7cf6632d4395\") " pod="ai-tenant-e2e-inf-b-24873d/test-model-24873d-kserve-dff86779c-29lrm"
Sep 14 10:01:27.426772 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:27.426723    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d55a291-49c9-4c1e-8ae1-7cf6632d4395-dshm\") pod \"test-model-24873d-kserve-dff86779c-29lrm\" (UID: \"0d55a291-49c9-4c1e-8ae1-7cf6632d4395\") " pod="ai-tenant-e2e-inf-b-24873d/test-model-24873d-kserve-dff86779c-29lrm"
Sep 14 10:01:27.427026 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:27.426843    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d55a291-49c9-4c1e-8ae1-7cf6632d4395-tmp-dir\") pod \"test-model-24873d-kserve-dff86779c-29lrm\" (UID: \"0d55a291-49c9-4c1e-8ae1-7cf6632d4395\") " pod="ai-tenant-e2e-inf-b-24873d/test-model-24873d-kserve-dff86779c-29lrm"
Sep 14 10:01:27.427026 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:27.426899    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d55a291-49c9-4c1e-8ae1-7cf6632d4395-tls-certs\") pod \"test-model-24873d-kserve-dff86779c-29lrm\" (UID: \"0d55a291-49c9-4c1e-8ae1-7cf6632d4395\") " pod="ai-tenant-e2e-inf-b-24873d/test-model-24873d-kserve-dff86779c-29lrm"
Sep 14 10:01:27.427026 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:27.426939    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d55a291-49c9-4c1e-8ae1-7cf6632d4395-home\") pod \"test-model-24873d-kserve-dff86779c-29lrm\" (UID: \"0d55a291-49c9-4c1e-8ae1-7cf6632d4395\") " pod="ai-tenant-e2e-inf-b-24873d/test-model-24873d-kserve-dff86779c-29lrm"
Sep 14 10:01:27.427026 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:27.426968    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx88v\" (UniqueName: \"kubernetes.io/projected/0d55a291-49c9-4c1e-8ae1-7cf6632d4395-kube-api-access-gx88v\") pod \"test-model-24873d-kserve-dff86779c-29lrm\" (UID: \"0d55a291-49c9-4c1e-8ae1-7cf6632d4395\") " pod="ai-tenant-e2e-inf-b-24873d/test-model-24873d-kserve-dff86779c-29lrm"
Sep 14 10:01:27.427313 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:27.427203    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d55a291-49c9-4c1e-8ae1-7cf6632d4395-model-cache\") pod \"test-model-24873d-kserve-dff86779c-29lrm\" (UID: \"0d55a291-49c9-4c1e-8ae1-7cf6632d4395\") " pod="ai-tenant-e2e-inf-b-24873d/test-model-24873d-kserve-dff86779c-29lrm"
Sep 14 10:01:27.427313 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:27.427289    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d55a291-49c9-4c1e-8ae1-7cf6632d4395-tmp-dir\") pod \"test-model-24873d-kserve-dff86779c-29lrm\" (UID: \"0d55a291-49c9-4c1e-8ae1-7cf6632d4395\") " pod="ai-tenant-e2e-inf-b-24873d/test-model-24873d-kserve-dff86779c-29lrm"
Sep 14 10:01:27.427403 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:27.427371    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d55a291-49c9-4c1e-8ae1-7cf6632d4395-home\") pod \"test-model-24873d-kserve-dff86779c-29lrm\" (UID: \"0d55a291-49c9-4c1e-8ae1-7cf6632d4395\") " pod="ai-tenant-e2e-inf-b-24873d/test-model-24873d-kserve-dff86779c-29lrm"
Sep 14 10:01:27.429157 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:27.429134    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d55a291-49c9-4c1e-8ae1-7cf6632d4395-dshm\") pod \"test-model-24873d-kserve-dff86779c-29lrm\" (UID: \"0d55a291-49c9-4c1e-8ae1-7cf6632d4395\") " pod="ai-tenant-e2e-inf-b-24873d/test-model-24873d-kserve-dff86779c-29lrm"
Sep 14 10:01:27.429442 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:27.429424    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d55a291-49c9-4c1e-8ae1-7cf6632d4395-tls-certs\") pod \"test-model-24873d-kserve-dff86779c-29lrm\" (UID: \"0d55a291-49c9-4c1e-8ae1-7cf6632d4395\") " pod="ai-tenant-e2e-inf-b-24873d/test-model-24873d-kserve-dff86779c-29lrm"
Sep 14 10:01:27.441581 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:27.441539    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx88v\" (UniqueName: \"kubernetes.io/projected/0d55a291-49c9-4c1e-8ae1-7cf6632d4395-kube-api-access-gx88v\") pod \"test-model-24873d-kserve-dff86779c-29lrm\" (UID: \"0d55a291-49c9-4c1e-8ae1-7cf6632d4395\") " pod="ai-tenant-e2e-inf-b-24873d/test-model-24873d-kserve-dff86779c-29lrm"
Sep 14 10:01:27.485491 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:27.485450    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-24873d/test-model-24873d-kserve-dff86779c-29lrm"
Sep 14 10:01:27.703602 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:27.703508    2563 generic.go:356] "Generic (PLEG): container finished" podID="6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d" containerID="a73865c4e4cf5831364578f49708bffeb619db64eecfcae81eb51b77470e959c" exitCode=0
Sep 14 10:01:27.703602 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:27.703579    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48" event={"ID":"6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d","Type":"ContainerDied","Data":"a73865c4e4cf5831364578f49708bffeb619db64eecfcae81eb51b77470e959c"}
Sep 14 10:01:27.704159 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:27.704138    2563 scope.go:117] "RemoveContainer" containerID="a73865c4e4cf5831364578f49708bffeb619db64eecfcae81eb51b77470e959c"
Sep 14 10:01:27.705502 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:27.705469    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 10:01:27.705589 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:27.705548    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 10:01:27.705589 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:27.705581    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 10:01:27.795230 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:27.795180    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-24873d/test-model-24873d-kserve-dff86779c-29lrm"]
Sep 14 10:01:28.064052 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:28.064016    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:01:28.712294 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:28.712250    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48" event={"ID":"6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d","Type":"ContainerStarted","Data":"f0a0884c1395aee0f96026b91447e3cb861a911d6e9aa38bb13494e12697dc1d"}
Sep 14 10:01:28.714169 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:28.714133    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-24873d/test-model-24873d-kserve-dff86779c-29lrm" event={"ID":"0d55a291-49c9-4c1e-8ae1-7cf6632d4395","Type":"ContainerStarted","Data":"d3f5faa3bb17859a0b83bb52a0cf6346c61bf5e3b2b752b1e70c2c9d61c920ed"}
Sep 14 10:01:28.714328 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:28.714181    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-24873d/test-model-24873d-kserve-dff86779c-29lrm" event={"ID":"0d55a291-49c9-4c1e-8ae1-7cf6632d4395","Type":"ContainerStarted","Data":"eb883d87c4df91e19077fd77e4aa57cd91ce02dd902ddc94906c86948da35e44"}
Sep 14 10:01:28.764586 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:28.764526    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-24873d/test-model-24873d-kserve-dff86779c-29lrm" podStartSLOduration=1.764505558 podStartE2EDuration="1.764505558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:01:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:01:28.760357947 +0000 UTC m=+1077.800032568" watchObservedRunningTime="2026-09-14 10:01:28.764505558 +0000 UTC m=+1077.804180219"
Sep 14 10:01:28.984245 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:28.984216    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:01:29.210024 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:29.209956    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48"
Sep 14 10:01:29.211349 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:29.211320    2563 patch_prober.go:28] interesting pod/e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Sep 14 10:01:29.211486 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:29.211376    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48" podUID="6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Sep 14 10:01:29.844469 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:29.844418    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:01:30.210528 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:30.210430    2563 patch_prober.go:28] interesting pod/e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Sep 14 10:01:30.210528 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:30.210509    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48" podUID="6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Sep 14 10:01:30.499308 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:30.499268    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48"
Sep 14 10:01:30.718295 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:30.718258    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:01:31.209853 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:31.209816    2563 patch_prober.go:28] interesting pod/e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Sep 14 10:01:31.210317 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:31.209885    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48" podUID="6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Sep 14 10:01:31.609546 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:31.609514    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:01:31.759617 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:31.759574    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:01:32.209734 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:32.209690    2563 patch_prober.go:28] interesting pod/e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Sep 14 10:01:32.209924 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:32.209758    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48" podUID="6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Sep 14 10:01:32.634334 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:32.634195    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:01:33.209953 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:33.209914    2563 patch_prober.go:28] interesting pod/e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Sep 14 10:01:33.210390 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:33.209974    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48" podUID="6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Sep 14 10:01:33.664142 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:33.664090    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:01:33.977530 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:33.977431    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:01:34.209681 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:34.209629    2563 patch_prober.go:28] interesting pod/e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Sep 14 10:01:34.209869 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:34.209712    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48" podUID="6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Sep 14 10:01:35.089197 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:35.089149    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:01:35.165995 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:35.165959    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:01:35.214486 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:35.214448    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48"
Sep 14 10:01:35.215760 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:35.215732    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48"
Sep 14 10:01:35.761706 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:35.761659    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:01:37.486588 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:37.486546    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-24873d/test-model-24873d-kserve-dff86779c-29lrm"
Sep 14 10:01:37.486588 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:37.486602    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-24873d/test-model-24873d-kserve-dff86779c-29lrm"
Sep 14 10:01:37.499366 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:37.499334    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-24873d/test-model-24873d-kserve-dff86779c-29lrm"
Sep 14 10:01:37.806701 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:37.806616    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-24873d/test-model-24873d-kserve-dff86779c-29lrm"
Sep 14 10:01:42.231329 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:42.231291    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:01:42.381683 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:42.381644    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:01:42.491553 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:42.491486    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:01:42.742217 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:42.742116    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f"]
Sep 14 10:01:42.742555 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:42.742542    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eae8537b-56e1-4d2a-a43b-de5bb82f78b8" containerName="istio-proxy"
Sep 14 10:01:42.742604 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:42.742557    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="eae8537b-56e1-4d2a-a43b-de5bb82f78b8" containerName="istio-proxy"
Sep 14 10:01:42.745634 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:42.745611    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f"
Sep 14 10:01:42.747869 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:42.747839    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-bc008e9d-openshift-default-dockercfg-hjqrw\""
Sep 14 10:01:42.764612 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:42.764572    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f"]
Sep 14 10:01:42.785966 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:42.785913    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:01:42.795308 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:42.795268    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9470ce1f-5f66-492b-ac4e-a64e41e787f1-istio-token\") pod \"e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f\" (UID: \"9470ce1f-5f66-492b-ac4e-a64e41e787f1\") " pod="openshift-ingress/e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f"
Sep 14 10:01:42.795509 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:42.795317    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9470ce1f-5f66-492b-ac4e-a64e41e787f1-credential-socket\") pod \"e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f\" (UID: \"9470ce1f-5f66-492b-ac4e-a64e41e787f1\") " pod="openshift-ingress/e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f"
Sep 14 10:01:42.795509 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:42.795343    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9470ce1f-5f66-492b-ac4e-a64e41e787f1-workload-socket\") pod \"e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f\" (UID: \"9470ce1f-5f66-492b-ac4e-a64e41e787f1\") " pod="openshift-ingress/e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f"
Sep 14 10:01:42.795509 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:42.795404    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9470ce1f-5f66-492b-ac4e-a64e41e787f1-istio-data\") pod \"e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f\" (UID: \"9470ce1f-5f66-492b-ac4e-a64e41e787f1\") " pod="openshift-ingress/e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f"
Sep 14 10:01:42.795509 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:42.795423    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9470ce1f-5f66-492b-ac4e-a64e41e787f1-istio-podinfo\") pod \"e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f\" (UID: \"9470ce1f-5f66-492b-ac4e-a64e41e787f1\") " pod="openshift-ingress/e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f"
Sep 14 10:01:42.795509 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:42.795489    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hld8q\" (UniqueName: \"kubernetes.io/projected/9470ce1f-5f66-492b-ac4e-a64e41e787f1-kube-api-access-hld8q\") pod \"e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f\" (UID: \"9470ce1f-5f66-492b-ac4e-a64e41e787f1\") " pod="openshift-ingress/e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f"
Sep 14 10:01:42.795767 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:42.795511    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9470ce1f-5f66-492b-ac4e-a64e41e787f1-workload-certs\") pod \"e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f\" (UID: \"9470ce1f-5f66-492b-ac4e-a64e41e787f1\") " pod="openshift-ingress/e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f"
Sep 14 10:01:42.795767 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:42.795649    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9470ce1f-5f66-492b-ac4e-a64e41e787f1-istiod-ca-cert\") pod \"e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f\" (UID: \"9470ce1f-5f66-492b-ac4e-a64e41e787f1\") " pod="openshift-ingress/e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f"
Sep 14 10:01:42.795767 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:42.795734    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9470ce1f-5f66-492b-ac4e-a64e41e787f1-istio-envoy\") pod \"e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f\" (UID: \"9470ce1f-5f66-492b-ac4e-a64e41e787f1\") " pod="openshift-ingress/e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f"
Sep 14 10:01:42.896816 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:42.896776    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9470ce1f-5f66-492b-ac4e-a64e41e787f1-istiod-ca-cert\") pod \"e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f\" (UID: \"9470ce1f-5f66-492b-ac4e-a64e41e787f1\") " pod="openshift-ingress/e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f"
Sep 14 10:01:42.897017 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:42.896838    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9470ce1f-5f66-492b-ac4e-a64e41e787f1-istio-envoy\") pod \"e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f\" (UID: \"9470ce1f-5f66-492b-ac4e-a64e41e787f1\") " pod="openshift-ingress/e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f"
Sep 14 10:01:42.897017 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:42.896860    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9470ce1f-5f66-492b-ac4e-a64e41e787f1-istio-token\") pod \"e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f\" (UID: \"9470ce1f-5f66-492b-ac4e-a64e41e787f1\") " pod="openshift-ingress/e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f"
Sep 14 10:01:42.897017 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:42.896884    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9470ce1f-5f66-492b-ac4e-a64e41e787f1-credential-socket\") pod \"e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f\" (UID: \"9470ce1f-5f66-492b-ac4e-a64e41e787f1\") " pod="openshift-ingress/e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f"
Sep 14 10:01:42.897017 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:42.896906    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9470ce1f-5f66-492b-ac4e-a64e41e787f1-workload-socket\") pod \"e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f\" (UID: \"9470ce1f-5f66-492b-ac4e-a64e41e787f1\") " pod="openshift-ingress/e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f"
Sep 14 10:01:42.897254 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:42.897141    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9470ce1f-5f66-492b-ac4e-a64e41e787f1-istio-data\") pod \"e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f\" (UID: \"9470ce1f-5f66-492b-ac4e-a64e41e787f1\") " pod="openshift-ingress/e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f"
Sep 14 10:01:42.897254 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:42.897181    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9470ce1f-5f66-492b-ac4e-a64e41e787f1-istio-podinfo\") pod \"e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f\" (UID: \"9470ce1f-5f66-492b-ac4e-a64e41e787f1\") " pod="openshift-ingress/e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f"
Sep 14 10:01:42.897254 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:42.897240    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hld8q\" (UniqueName: \"kubernetes.io/projected/9470ce1f-5f66-492b-ac4e-a64e41e787f1-kube-api-access-hld8q\") pod \"e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f\" (UID: \"9470ce1f-5f66-492b-ac4e-a64e41e787f1\") " pod="openshift-ingress/e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f"
Sep 14 10:01:42.897423 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:42.897263    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9470ce1f-5f66-492b-ac4e-a64e41e787f1-workload-certs\") pod \"e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f\" (UID: \"9470ce1f-5f66-492b-ac4e-a64e41e787f1\") " pod="openshift-ingress/e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f"
Sep 14 10:01:42.897538 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:42.897515    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9470ce1f-5f66-492b-ac4e-a64e41e787f1-workload-socket\") pod \"e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f\" (UID: \"9470ce1f-5f66-492b-ac4e-a64e41e787f1\") " pod="openshift-ingress/e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f"
Sep 14 10:01:42.897725 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:42.897624    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9470ce1f-5f66-492b-ac4e-a64e41e787f1-istio-data\") pod \"e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f\" (UID: \"9470ce1f-5f66-492b-ac4e-a64e41e787f1\") " pod="openshift-ingress/e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f"
Sep 14 10:01:42.897725 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:42.897691    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9470ce1f-5f66-492b-ac4e-a64e41e787f1-credential-socket\") pod \"e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f\" (UID: \"9470ce1f-5f66-492b-ac4e-a64e41e787f1\") " pod="openshift-ingress/e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f"
Sep 14 10:01:42.897725 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:42.897712    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9470ce1f-5f66-492b-ac4e-a64e41e787f1-istiod-ca-cert\") pod \"e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f\" (UID: \"9470ce1f-5f66-492b-ac4e-a64e41e787f1\") " pod="openshift-ingress/e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f"
Sep 14 10:01:42.897877 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:42.897768    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9470ce1f-5f66-492b-ac4e-a64e41e787f1-workload-certs\") pod \"e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f\" (UID: \"9470ce1f-5f66-492b-ac4e-a64e41e787f1\") " pod="openshift-ingress/e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f"
Sep 14 10:01:42.899359 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:42.899338    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9470ce1f-5f66-492b-ac4e-a64e41e787f1-istio-envoy\") pod \"e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f\" (UID: \"9470ce1f-5f66-492b-ac4e-a64e41e787f1\") " pod="openshift-ingress/e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f"
Sep 14 10:01:42.899627 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:42.899609    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9470ce1f-5f66-492b-ac4e-a64e41e787f1-istio-podinfo\") pod \"e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f\" (UID: \"9470ce1f-5f66-492b-ac4e-a64e41e787f1\") " pod="openshift-ingress/e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f"
Sep 14 10:01:42.907928 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:42.907894    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9470ce1f-5f66-492b-ac4e-a64e41e787f1-istio-token\") pod \"e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f\" (UID: \"9470ce1f-5f66-492b-ac4e-a64e41e787f1\") " pod="openshift-ingress/e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f"
Sep 14 10:01:42.908209 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:42.908032    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hld8q\" (UniqueName: \"kubernetes.io/projected/9470ce1f-5f66-492b-ac4e-a64e41e787f1-kube-api-access-hld8q\") pod \"e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f\" (UID: \"9470ce1f-5f66-492b-ac4e-a64e41e787f1\") " pod="openshift-ingress/e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f"
Sep 14 10:01:43.057402 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:43.057293    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f"
Sep 14 10:01:43.396345 ip-10-0-132-4 kubenswrapper[2563]: W0914 10:01:43.396310    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9470ce1f_5f66_492b_ac4e_a64e41e787f1.slice/crio-7e0700a914edda6956bda38e8af5d6f7f8c02f5903f37cd490ae31bfd1a23d09 WatchSource:0}: Error finding container 7e0700a914edda6956bda38e8af5d6f7f8c02f5903f37cd490ae31bfd1a23d09: Status 404 returned error can't find the container with id 7e0700a914edda6956bda38e8af5d6f7f8c02f5903f37cd490ae31bfd1a23d09
Sep 14 10:01:43.396717 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:43.396692    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f"]
Sep 14 10:01:43.398948 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:43.398908    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 10:01:43.399058 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:43.398993    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 10:01:43.399058 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:43.399055    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 10:01:43.540959 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:43.540914    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:01:43.839065 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:43.839026    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f" event={"ID":"9470ce1f-5f66-492b-ac4e-a64e41e787f1","Type":"ContainerStarted","Data":"2fa2e9c50435513b7fa25e375bfca295b74eee1a87b076cb6478e50a2c3db991"}
Sep 14 10:01:43.839281 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:43.839092    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f" event={"ID":"9470ce1f-5f66-492b-ac4e-a64e41e787f1","Type":"ContainerStarted","Data":"7e0700a914edda6956bda38e8af5d6f7f8c02f5903f37cd490ae31bfd1a23d09"}
Sep 14 10:01:43.865678 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:43.865618    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f" podStartSLOduration=1.865596043 podStartE2EDuration="1.865596043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:01:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:01:43.861527037 +0000 UTC m=+1092.901201660" watchObservedRunningTime="2026-09-14 10:01:43.865596043 +0000 UTC m=+1092.905270664"
Sep 14 10:01:44.057721 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:44.057683    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f"
Sep 14 10:01:44.063350 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:44.063317    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f"
Sep 14 10:01:44.845058 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:44.844996    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f"
Sep 14 10:01:44.846106 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:44.846083    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f"
Sep 14 10:01:48.400295 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:48.400256    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:01:49.101154 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:49.101108    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:01:49.717542 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:49.717490    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:01:49.824211 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:49.824170    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:01:50.019877 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:50.019830    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:01:51.297781 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:51.297740    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:01:52.260571 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:52.260530    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:01:52.811402 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:52.811362    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:01:54.483957 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:54.483918    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:01:54.894967 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:54.894930    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:01:55.946389 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:55.946356    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:01:56.055260 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:56.055222    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:01:56.509453 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:56.509418    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:01:56.610518 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:01:56.610482    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:02:15.887289 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:15.887231    2563 patch_prober.go:28] interesting pod/e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 14 10:02:15.887740 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:15.887340    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f" podUID="9470ce1f-5f66-492b-ac4e-a64e41e787f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 14 10:02:17.079033 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:17.078917    2563 generic.go:356] "Generic (PLEG): container finished" podID="9470ce1f-5f66-492b-ac4e-a64e41e787f1" containerID="2fa2e9c50435513b7fa25e375bfca295b74eee1a87b076cb6478e50a2c3db991" exitCode=0
Sep 14 10:02:17.079433 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:17.078994    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f" event={"ID":"9470ce1f-5f66-492b-ac4e-a64e41e787f1","Type":"ContainerDied","Data":"2fa2e9c50435513b7fa25e375bfca295b74eee1a87b076cb6478e50a2c3db991"}
Sep 14 10:02:17.079476 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:17.079457    2563 scope.go:117] "RemoveContainer" containerID="2fa2e9c50435513b7fa25e375bfca295b74eee1a87b076cb6478e50a2c3db991"
Sep 14 10:02:17.080774 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:17.080739    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 10:02:17.080840 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:17.080821    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 10:02:17.080883 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:17.080853    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 10:02:18.087793 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:18.087751    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f" event={"ID":"9470ce1f-5f66-492b-ac4e-a64e41e787f1","Type":"ContainerStarted","Data":"912edec05413cc804221688cf327c78beb3feb5c956c643a3b7c28b7892a7a20"}
Sep 14 10:02:18.476212 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:18.476063    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:02:19.058086 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:19.058045    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f"
Sep 14 10:02:19.058366 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:19.058341    2563 patch_prober.go:28] interesting pod/e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Sep 14 10:02:19.058418 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:19.058397    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f" podUID="9470ce1f-5f66-492b-ac4e-a64e41e787f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Sep 14 10:02:19.446714 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:19.446624    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:02:19.648359 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:19.648327    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:02:20.058045 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:20.057987    2563 patch_prober.go:28] interesting pod/e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Sep 14 10:02:20.058249 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:20.058076    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f" podUID="9470ce1f-5f66-492b-ac4e-a64e41e787f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Sep 14 10:02:20.263844 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:20.263804    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:02:20.373490 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:20.373398    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:02:20.752513 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:20.752475    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:02:21.058340 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:21.058243    2563 patch_prober.go:28] interesting pod/e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Sep 14 10:02:21.058340 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:21.058326    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f" podUID="9470ce1f-5f66-492b-ac4e-a64e41e787f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Sep 14 10:02:22.058173 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:22.058131    2563 patch_prober.go:28] interesting pod/e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Sep 14 10:02:22.058598 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:22.058211    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f" podUID="9470ce1f-5f66-492b-ac4e-a64e41e787f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Sep 14 10:02:22.286526 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:22.286485    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:02:22.402477 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:22.402382    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:02:23.058437 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:23.058397    2563 patch_prober.go:28] interesting pod/e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Sep 14 10:02:23.058936 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:23.058470    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f" podUID="9470ce1f-5f66-492b-ac4e-a64e41e787f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Sep 14 10:02:23.100157 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:23.100102    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-31f3b6/explicit-model-31f3b6-kserve-59cc9f967f-4sfx5"]
Sep 14 10:02:23.102820 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:23.102796    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-31f3b6/explicit-model-31f3b6-kserve-59cc9f967f-4sfx5"
Sep 14 10:02:23.106443 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:23.106411    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-31f3b6\"/\"default-dockercfg-l4c58\""
Sep 14 10:02:23.106629 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:23.106442    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-31f3b6\"/\"explicit-model-31f3b6-kserve-self-signed-certs\""
Sep 14 10:02:23.106771 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:23.106749    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-31f3b6\"/\"kube-root-ca.crt\""
Sep 14 10:02:23.106885 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:23.106811    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-31f3b6\"/\"openshift-service-ca.crt\""
Sep 14 10:02:23.147720 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:23.147631    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:02:23.154208 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:23.154130    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-31f3b6/explicit-model-31f3b6-kserve-59cc9f967f-4sfx5"]
Sep 14 10:02:23.253030 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:23.252639    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85054bf7-c54c-40aa-9693-8360bfd9b5cb-home\") pod \"explicit-model-31f3b6-kserve-59cc9f967f-4sfx5\" (UID: \"85054bf7-c54c-40aa-9693-8360bfd9b5cb\") " pod="ai-tenant-e2e-explicit-31f3b6/explicit-model-31f3b6-kserve-59cc9f967f-4sfx5"
Sep 14 10:02:23.253030 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:23.252700    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85054bf7-c54c-40aa-9693-8360bfd9b5cb-dshm\") pod \"explicit-model-31f3b6-kserve-59cc9f967f-4sfx5\" (UID: \"85054bf7-c54c-40aa-9693-8360bfd9b5cb\") " pod="ai-tenant-e2e-explicit-31f3b6/explicit-model-31f3b6-kserve-59cc9f967f-4sfx5"
Sep 14 10:02:23.253030 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:23.252768    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k9m8\" (UniqueName: \"kubernetes.io/projected/85054bf7-c54c-40aa-9693-8360bfd9b5cb-kube-api-access-2k9m8\") pod \"explicit-model-31f3b6-kserve-59cc9f967f-4sfx5\" (UID: \"85054bf7-c54c-40aa-9693-8360bfd9b5cb\") " pod="ai-tenant-e2e-explicit-31f3b6/explicit-model-31f3b6-kserve-59cc9f967f-4sfx5"
Sep 14 10:02:23.253030 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:23.252831    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85054bf7-c54c-40aa-9693-8360bfd9b5cb-tmp-dir\") pod \"explicit-model-31f3b6-kserve-59cc9f967f-4sfx5\" (UID: \"85054bf7-c54c-40aa-9693-8360bfd9b5cb\") " pod="ai-tenant-e2e-explicit-31f3b6/explicit-model-31f3b6-kserve-59cc9f967f-4sfx5"
Sep 14 10:02:23.253030 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:23.252894    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85054bf7-c54c-40aa-9693-8360bfd9b5cb-tls-certs\") pod \"explicit-model-31f3b6-kserve-59cc9f967f-4sfx5\" (UID: \"85054bf7-c54c-40aa-9693-8360bfd9b5cb\") " pod="ai-tenant-e2e-explicit-31f3b6/explicit-model-31f3b6-kserve-59cc9f967f-4sfx5"
Sep 14 10:02:23.253030 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:23.252945    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85054bf7-c54c-40aa-9693-8360bfd9b5cb-model-cache\") pod \"explicit-model-31f3b6-kserve-59cc9f967f-4sfx5\" (UID: \"85054bf7-c54c-40aa-9693-8360bfd9b5cb\") " pod="ai-tenant-e2e-explicit-31f3b6/explicit-model-31f3b6-kserve-59cc9f967f-4sfx5"
Sep 14 10:02:23.354150 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:23.354047    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85054bf7-c54c-40aa-9693-8360bfd9b5cb-home\") pod \"explicit-model-31f3b6-kserve-59cc9f967f-4sfx5\" (UID: \"85054bf7-c54c-40aa-9693-8360bfd9b5cb\") " pod="ai-tenant-e2e-explicit-31f3b6/explicit-model-31f3b6-kserve-59cc9f967f-4sfx5"
Sep 14 10:02:23.354150 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:23.354085    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85054bf7-c54c-40aa-9693-8360bfd9b5cb-dshm\") pod \"explicit-model-31f3b6-kserve-59cc9f967f-4sfx5\" (UID: \"85054bf7-c54c-40aa-9693-8360bfd9b5cb\") " pod="ai-tenant-e2e-explicit-31f3b6/explicit-model-31f3b6-kserve-59cc9f967f-4sfx5"
Sep 14 10:02:23.354150 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:23.354128    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k9m8\" (UniqueName: \"kubernetes.io/projected/85054bf7-c54c-40aa-9693-8360bfd9b5cb-kube-api-access-2k9m8\") pod \"explicit-model-31f3b6-kserve-59cc9f967f-4sfx5\" (UID: \"85054bf7-c54c-40aa-9693-8360bfd9b5cb\") " pod="ai-tenant-e2e-explicit-31f3b6/explicit-model-31f3b6-kserve-59cc9f967f-4sfx5"
Sep 14 10:02:23.354459 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:23.354191    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85054bf7-c54c-40aa-9693-8360bfd9b5cb-tmp-dir\") pod \"explicit-model-31f3b6-kserve-59cc9f967f-4sfx5\" (UID: \"85054bf7-c54c-40aa-9693-8360bfd9b5cb\") " pod="ai-tenant-e2e-explicit-31f3b6/explicit-model-31f3b6-kserve-59cc9f967f-4sfx5"
Sep 14 10:02:23.354459 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:23.354241    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85054bf7-c54c-40aa-9693-8360bfd9b5cb-tls-certs\") pod \"explicit-model-31f3b6-kserve-59cc9f967f-4sfx5\" (UID: \"85054bf7-c54c-40aa-9693-8360bfd9b5cb\") " pod="ai-tenant-e2e-explicit-31f3b6/explicit-model-31f3b6-kserve-59cc9f967f-4sfx5"
Sep 14 10:02:23.354459 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:23.354278    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85054bf7-c54c-40aa-9693-8360bfd9b5cb-model-cache\") pod \"explicit-model-31f3b6-kserve-59cc9f967f-4sfx5\" (UID: \"85054bf7-c54c-40aa-9693-8360bfd9b5cb\") " pod="ai-tenant-e2e-explicit-31f3b6/explicit-model-31f3b6-kserve-59cc9f967f-4sfx5"
Sep 14 10:02:23.354597 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:23.354578    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85054bf7-c54c-40aa-9693-8360bfd9b5cb-home\") pod \"explicit-model-31f3b6-kserve-59cc9f967f-4sfx5\" (UID: \"85054bf7-c54c-40aa-9693-8360bfd9b5cb\") " pod="ai-tenant-e2e-explicit-31f3b6/explicit-model-31f3b6-kserve-59cc9f967f-4sfx5"
Sep 14 10:02:23.354639 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:23.354598    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85054bf7-c54c-40aa-9693-8360bfd9b5cb-tmp-dir\") pod \"explicit-model-31f3b6-kserve-59cc9f967f-4sfx5\" (UID: \"85054bf7-c54c-40aa-9693-8360bfd9b5cb\") " pod="ai-tenant-e2e-explicit-31f3b6/explicit-model-31f3b6-kserve-59cc9f967f-4sfx5"
Sep 14 10:02:23.354675 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:23.354657    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85054bf7-c54c-40aa-9693-8360bfd9b5cb-model-cache\") pod \"explicit-model-31f3b6-kserve-59cc9f967f-4sfx5\" (UID: \"85054bf7-c54c-40aa-9693-8360bfd9b5cb\") " pod="ai-tenant-e2e-explicit-31f3b6/explicit-model-31f3b6-kserve-59cc9f967f-4sfx5"
Sep 14 10:02:23.356661 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:23.356635    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85054bf7-c54c-40aa-9693-8360bfd9b5cb-dshm\") pod \"explicit-model-31f3b6-kserve-59cc9f967f-4sfx5\" (UID: \"85054bf7-c54c-40aa-9693-8360bfd9b5cb\") " pod="ai-tenant-e2e-explicit-31f3b6/explicit-model-31f3b6-kserve-59cc9f967f-4sfx5"
Sep 14 10:02:23.356820 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:23.356800    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85054bf7-c54c-40aa-9693-8360bfd9b5cb-tls-certs\") pod \"explicit-model-31f3b6-kserve-59cc9f967f-4sfx5\" (UID: \"85054bf7-c54c-40aa-9693-8360bfd9b5cb\") " pod="ai-tenant-e2e-explicit-31f3b6/explicit-model-31f3b6-kserve-59cc9f967f-4sfx5"
Sep 14 10:02:23.363812 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:23.363771    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k9m8\" (UniqueName: \"kubernetes.io/projected/85054bf7-c54c-40aa-9693-8360bfd9b5cb-kube-api-access-2k9m8\") pod \"explicit-model-31f3b6-kserve-59cc9f967f-4sfx5\" (UID: \"85054bf7-c54c-40aa-9693-8360bfd9b5cb\") " pod="ai-tenant-e2e-explicit-31f3b6/explicit-model-31f3b6-kserve-59cc9f967f-4sfx5"
Sep 14 10:02:23.414819 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:23.414780    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-31f3b6/explicit-model-31f3b6-kserve-59cc9f967f-4sfx5"
Sep 14 10:02:23.484443 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:23.484398    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:02:23.736108 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:23.736026    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-31f3b6/explicit-model-31f3b6-kserve-59cc9f967f-4sfx5"]
Sep 14 10:02:23.751332 ip-10-0-132-4 kubenswrapper[2563]: W0914 10:02:23.751265    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod85054bf7_c54c_40aa_9693_8360bfd9b5cb.slice/crio-c322d97bb133cbba8897dc88933fa895642fdd97b0bed149fa9a761ce9755d11 WatchSource:0}: Error finding container c322d97bb133cbba8897dc88933fa895642fdd97b0bed149fa9a761ce9755d11: Status 404 returned error can't find the container with id c322d97bb133cbba8897dc88933fa895642fdd97b0bed149fa9a761ce9755d11
Sep 14 10:02:24.062527 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:24.062443    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f"
Sep 14 10:02:24.062900 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:24.062844    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f"
Sep 14 10:02:24.063808 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:24.063790    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f"
Sep 14 10:02:24.136271 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:24.136218    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-31f3b6/explicit-model-31f3b6-kserve-59cc9f967f-4sfx5" event={"ID":"85054bf7-c54c-40aa-9693-8360bfd9b5cb","Type":"ContainerStarted","Data":"21dc3730d914e5003becddbf45e2ff06baebb8cc0dc78d92364e2815dee5a587"}
Sep 14 10:02:24.136488 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:24.136285    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-31f3b6/explicit-model-31f3b6-kserve-59cc9f967f-4sfx5" event={"ID":"85054bf7-c54c-40aa-9693-8360bfd9b5cb","Type":"ContainerStarted","Data":"c322d97bb133cbba8897dc88933fa895642fdd97b0bed149fa9a761ce9755d11"}
Sep 14 10:02:24.172976 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:24.172915    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-31f3b6/explicit-model-31f3b6-kserve-59cc9f967f-4sfx5" podStartSLOduration=1.172898163 podStartE2EDuration="1.172898163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:02:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:02:24.169430335 +0000 UTC m=+1133.209104957" watchObservedRunningTime="2026-09-14 10:02:24.172898163 +0000 UTC m=+1133.212572876"
Sep 14 10:02:24.525340 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:24.525292    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:02:27.305914 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:27.305875    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:02:27.413900 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:27.413852    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:02:27.682487 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:27.682393    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bc008e9d-6gggx"]
Sep 14 10:02:27.685254 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:27.685229    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bc008e9d-6gggx"
Sep 14 10:02:27.687516 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:27.687487    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 14 10:02:27.687667 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:27.687555    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gf4xj\""
Sep 14 10:02:27.694404 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:27.694279    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bc008e9d-6gggx"]
Sep 14 10:02:27.806203 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:27.806161    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/148f5643-7d73-41b4-8f95-186b86ea68f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-bc008e9d-6gggx\" (UID: \"148f5643-7d73-41b4-8f95-186b86ea68f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bc008e9d-6gggx"
Sep 14 10:02:27.907613 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:27.907565    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/148f5643-7d73-41b4-8f95-186b86ea68f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-bc008e9d-6gggx\" (UID: \"148f5643-7d73-41b4-8f95-186b86ea68f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bc008e9d-6gggx"
Sep 14 10:02:27.908328 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:27.908305    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/148f5643-7d73-41b4-8f95-186b86ea68f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-bc008e9d-6gggx\" (UID: \"148f5643-7d73-41b4-8f95-186b86ea68f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bc008e9d-6gggx"
Sep 14 10:02:27.998128 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:27.998090    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bc008e9d-6gggx"
Sep 14 10:02:28.278664 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:28.278634    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bc008e9d-6gggx"]
Sep 14 10:02:28.280932 ip-10-0-132-4 kubenswrapper[2563]: W0914 10:02:28.280900    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod148f5643_7d73_41b4_8f95_186b86ea68f2.slice/crio-d23f522a2d9daadd9b880d2beaab787b516ebd8e12345088463c6f39c015fcba WatchSource:0}: Error finding container d23f522a2d9daadd9b880d2beaab787b516ebd8e12345088463c6f39c015fcba: Status 404 returned error can't find the container with id d23f522a2d9daadd9b880d2beaab787b516ebd8e12345088463c6f39c015fcba
Sep 14 10:02:28.631171 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:28.631077    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:02:29.171068 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:29.170991    2563 generic.go:356] "Generic (PLEG): container finished" podID="148f5643-7d73-41b4-8f95-186b86ea68f2" containerID="7b4e67cb3ca24e3bb1716a6ff04607c4c2fd3f61c4ae13533758c511e0373103" exitCode=0
Sep 14 10:02:29.171270 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:29.171073    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bc008e9d-6gggx" event={"ID":"148f5643-7d73-41b4-8f95-186b86ea68f2","Type":"ContainerDied","Data":"7b4e67cb3ca24e3bb1716a6ff04607c4c2fd3f61c4ae13533758c511e0373103"}
Sep 14 10:02:29.171270 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:29.171156    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bc008e9d-6gggx" event={"ID":"148f5643-7d73-41b4-8f95-186b86ea68f2","Type":"ContainerStarted","Data":"d23f522a2d9daadd9b880d2beaab787b516ebd8e12345088463c6f39c015fcba"}
Sep 14 10:02:29.694536 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:29.694495    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:02:30.323618 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:30.323579    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bc008e9d-6gggx"
Sep 14 10:02:30.437469 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:30.437370    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/148f5643-7d73-41b4-8f95-186b86ea68f2-maas-api-service-ca\") pod \"148f5643-7d73-41b4-8f95-186b86ea68f2\" (UID: \"148f5643-7d73-41b4-8f95-186b86ea68f2\") "
Sep 14 10:02:30.437780 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:30.437755    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/148f5643-7d73-41b4-8f95-186b86ea68f2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "148f5643-7d73-41b4-8f95-186b86ea68f2" (UID: "148f5643-7d73-41b4-8f95-186b86ea68f2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:02:30.538738 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:30.538686    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/148f5643-7d73-41b4-8f95-186b86ea68f2-maas-api-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:02:30.617904 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:30.617871    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:02:31.185841 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:31.185814    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bc008e9d-6gggx"
Sep 14 10:02:31.186276 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:31.185813    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bc008e9d-6gggx" event={"ID":"148f5643-7d73-41b4-8f95-186b86ea68f2","Type":"ContainerDied","Data":"d23f522a2d9daadd9b880d2beaab787b516ebd8e12345088463c6f39c015fcba"}
Sep 14 10:02:31.186276 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:31.185937    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d23f522a2d9daadd9b880d2beaab787b516ebd8e12345088463c6f39c015fcba"
Sep 14 10:02:31.244381 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:31.244339    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-31f3b6/explicit-model-31f3b6-kserve-59cc9f967f-4sfx5"]
Sep 14 10:02:31.266900 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:31.266866    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:02:31.354350 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:31.354307    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-31f3b6-9k6ng"]
Sep 14 10:02:31.354818 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:31.354799    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="148f5643-7d73-41b4-8f95-186b86ea68f2" containerName="revoke-keys"
Sep 14 10:02:31.354893 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:31.354820    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="148f5643-7d73-41b4-8f95-186b86ea68f2" containerName="revoke-keys"
Sep 14 10:02:31.354931 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:31.354906    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="148f5643-7d73-41b4-8f95-186b86ea68f2" containerName="revoke-keys"
Sep 14 10:02:31.357880 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:31.357857    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-31f3b6-9k6ng"
Sep 14 10:02:31.360065 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:31.360034    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 14 10:02:31.360229 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:31.360152    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gf4xj\""
Sep 14 10:02:31.369229 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:31.369178    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-31f3b6-9k6ng"]
Sep 14 10:02:31.449417 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:31.449307    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b670cf8e-480f-4f68-8b9f-50b2abb9dcea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-31f3b6-9k6ng\" (UID: \"b670cf8e-480f-4f68-8b9f-50b2abb9dcea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-31f3b6-9k6ng"
Sep 14 10:02:31.550301 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:31.550256    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b670cf8e-480f-4f68-8b9f-50b2abb9dcea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-31f3b6-9k6ng\" (UID: \"b670cf8e-480f-4f68-8b9f-50b2abb9dcea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-31f3b6-9k6ng"
Sep 14 10:02:31.550952 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:31.550931    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b670cf8e-480f-4f68-8b9f-50b2abb9dcea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-31f3b6-9k6ng\" (UID: \"b670cf8e-480f-4f68-8b9f-50b2abb9dcea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-31f3b6-9k6ng"
Sep 14 10:02:31.673866 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:31.673828    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-31f3b6-9k6ng"
Sep 14 10:02:31.704664 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:31.704566    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:02:31.950314 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:31.950283    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-31f3b6-9k6ng"]
Sep 14 10:02:31.951536 ip-10-0-132-4 kubenswrapper[2563]: W0914 10:02:31.951506    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb670cf8e_480f_4f68_8b9f_50b2abb9dcea.slice/crio-64b30ebeada7107bf15d24594490f3427afaf7c0a4770af316b0b6098c7bc61e WatchSource:0}: Error finding container 64b30ebeada7107bf15d24594490f3427afaf7c0a4770af316b0b6098c7bc61e: Status 404 returned error can't find the container with id 64b30ebeada7107bf15d24594490f3427afaf7c0a4770af316b0b6098c7bc61e
Sep 14 10:02:32.194194 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:32.194156    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-31f3b6-9k6ng" event={"ID":"b670cf8e-480f-4f68-8b9f-50b2abb9dcea","Type":"ContainerStarted","Data":"64b30ebeada7107bf15d24594490f3427afaf7c0a4770af316b0b6098c7bc61e"}
Sep 14 10:02:32.606857 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:32.606811    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:02:33.202779 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:33.202739    2563 generic.go:356] "Generic (PLEG): container finished" podID="b670cf8e-480f-4f68-8b9f-50b2abb9dcea" containerID="134a848791882da7d32205c1caad532204b88096609f6345e3b871cc6864ea96" exitCode=0
Sep 14 10:02:33.203219 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:33.202803    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-31f3b6-9k6ng" event={"ID":"b670cf8e-480f-4f68-8b9f-50b2abb9dcea","Type":"ContainerDied","Data":"134a848791882da7d32205c1caad532204b88096609f6345e3b871cc6864ea96"}
Sep 14 10:02:33.415885 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:33.415840    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-31f3b6/explicit-model-31f3b6-kserve-59cc9f967f-4sfx5"
Sep 14 10:02:33.549820 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:33.549776    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:02:34.316604 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:34.316548    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:02:34.353527 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:34.353494    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-31f3b6-9k6ng"
Sep 14 10:02:34.383669 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:34.383632    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b670cf8e-480f-4f68-8b9f-50b2abb9dcea-maas-api-service-ca\") pod \"b670cf8e-480f-4f68-8b9f-50b2abb9dcea\" (UID: \"b670cf8e-480f-4f68-8b9f-50b2abb9dcea\") "
Sep 14 10:02:34.384066 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:34.384045    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b670cf8e-480f-4f68-8b9f-50b2abb9dcea-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b670cf8e-480f-4f68-8b9f-50b2abb9dcea" (UID: "b670cf8e-480f-4f68-8b9f-50b2abb9dcea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:02:34.485543 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:34.485502    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b670cf8e-480f-4f68-8b9f-50b2abb9dcea-maas-api-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:02:34.756284 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:34.756196    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-2901ec/e2e-conflict-b-c109e646-kserve-7d9d8884d9-b6x76"]
Sep 14 10:02:34.756646 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:34.756632    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b670cf8e-480f-4f68-8b9f-50b2abb9dcea" containerName="revoke-keys"
Sep 14 10:02:34.756689 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:34.756648    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b670cf8e-480f-4f68-8b9f-50b2abb9dcea" containerName="revoke-keys"
Sep 14 10:02:34.756739 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:34.756707    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b670cf8e-480f-4f68-8b9f-50b2abb9dcea" containerName="revoke-keys"
Sep 14 10:02:34.759199 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:34.759180    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2901ec/e2e-conflict-b-c109e646-kserve-7d9d8884d9-b6x76"
Sep 14 10:02:34.761292 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:34.761264    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-2901ec\"/\"e2e-conflict-b-c109e646-kserve-self-signed-certs\""
Sep 14 10:02:34.770838 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:34.770803    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-2901ec/e2e-conflict-b-c109e646-kserve-7d9d8884d9-b6x76"]
Sep 14 10:02:34.780369 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:34.780338    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:02:34.788441 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:34.788406    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tb9t\" (UniqueName: \"kubernetes.io/projected/53af90b8-85ec-4fa4-9831-59e5e6ce0037-kube-api-access-8tb9t\") pod \"e2e-conflict-b-c109e646-kserve-7d9d8884d9-b6x76\" (UID: \"53af90b8-85ec-4fa4-9831-59e5e6ce0037\") " pod="e2e-models-e2e-worker-w4-2901ec/e2e-conflict-b-c109e646-kserve-7d9d8884d9-b6x76"
Sep 14 10:02:34.788441 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:34.788448    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53af90b8-85ec-4fa4-9831-59e5e6ce0037-model-cache\") pod \"e2e-conflict-b-c109e646-kserve-7d9d8884d9-b6x76\" (UID: \"53af90b8-85ec-4fa4-9831-59e5e6ce0037\") " pod="e2e-models-e2e-worker-w4-2901ec/e2e-conflict-b-c109e646-kserve-7d9d8884d9-b6x76"
Sep 14 10:02:34.788656 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:34.788477    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53af90b8-85ec-4fa4-9831-59e5e6ce0037-tls-certs\") pod \"e2e-conflict-b-c109e646-kserve-7d9d8884d9-b6x76\" (UID: \"53af90b8-85ec-4fa4-9831-59e5e6ce0037\") " pod="e2e-models-e2e-worker-w4-2901ec/e2e-conflict-b-c109e646-kserve-7d9d8884d9-b6x76"
Sep 14 10:02:34.788656 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:34.788517    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53af90b8-85ec-4fa4-9831-59e5e6ce0037-home\") pod \"e2e-conflict-b-c109e646-kserve-7d9d8884d9-b6x76\" (UID: \"53af90b8-85ec-4fa4-9831-59e5e6ce0037\") " pod="e2e-models-e2e-worker-w4-2901ec/e2e-conflict-b-c109e646-kserve-7d9d8884d9-b6x76"
Sep 14 10:02:34.788656 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:34.788552    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53af90b8-85ec-4fa4-9831-59e5e6ce0037-tmp-dir\") pod \"e2e-conflict-b-c109e646-kserve-7d9d8884d9-b6x76\" (UID: \"53af90b8-85ec-4fa4-9831-59e5e6ce0037\") " pod="e2e-models-e2e-worker-w4-2901ec/e2e-conflict-b-c109e646-kserve-7d9d8884d9-b6x76"
Sep 14 10:02:34.788656 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:34.788603    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53af90b8-85ec-4fa4-9831-59e5e6ce0037-dshm\") pod \"e2e-conflict-b-c109e646-kserve-7d9d8884d9-b6x76\" (UID: \"53af90b8-85ec-4fa4-9831-59e5e6ce0037\") " pod="e2e-models-e2e-worker-w4-2901ec/e2e-conflict-b-c109e646-kserve-7d9d8884d9-b6x76"
Sep 14 10:02:34.889962 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:34.889923    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53af90b8-85ec-4fa4-9831-59e5e6ce0037-home\") pod \"e2e-conflict-b-c109e646-kserve-7d9d8884d9-b6x76\" (UID: \"53af90b8-85ec-4fa4-9831-59e5e6ce0037\") " pod="e2e-models-e2e-worker-w4-2901ec/e2e-conflict-b-c109e646-kserve-7d9d8884d9-b6x76"
Sep 14 10:02:34.890195 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:34.889985    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53af90b8-85ec-4fa4-9831-59e5e6ce0037-tmp-dir\") pod \"e2e-conflict-b-c109e646-kserve-7d9d8884d9-b6x76\" (UID: \"53af90b8-85ec-4fa4-9831-59e5e6ce0037\") " pod="e2e-models-e2e-worker-w4-2901ec/e2e-conflict-b-c109e646-kserve-7d9d8884d9-b6x76"
Sep 14 10:02:34.890195 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:34.890089    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53af90b8-85ec-4fa4-9831-59e5e6ce0037-dshm\") pod \"e2e-conflict-b-c109e646-kserve-7d9d8884d9-b6x76\" (UID: \"53af90b8-85ec-4fa4-9831-59e5e6ce0037\") " pod="e2e-models-e2e-worker-w4-2901ec/e2e-conflict-b-c109e646-kserve-7d9d8884d9-b6x76"
Sep 14 10:02:34.890313 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:34.890214    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tb9t\" (UniqueName: \"kubernetes.io/projected/53af90b8-85ec-4fa4-9831-59e5e6ce0037-kube-api-access-8tb9t\") pod \"e2e-conflict-b-c109e646-kserve-7d9d8884d9-b6x76\" (UID: \"53af90b8-85ec-4fa4-9831-59e5e6ce0037\") " pod="e2e-models-e2e-worker-w4-2901ec/e2e-conflict-b-c109e646-kserve-7d9d8884d9-b6x76"
Sep 14 10:02:34.890313 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:34.890249    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53af90b8-85ec-4fa4-9831-59e5e6ce0037-model-cache\") pod \"e2e-conflict-b-c109e646-kserve-7d9d8884d9-b6x76\" (UID: \"53af90b8-85ec-4fa4-9831-59e5e6ce0037\") " pod="e2e-models-e2e-worker-w4-2901ec/e2e-conflict-b-c109e646-kserve-7d9d8884d9-b6x76"
Sep 14 10:02:34.890313 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:34.890295    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53af90b8-85ec-4fa4-9831-59e5e6ce0037-tls-certs\") pod \"e2e-conflict-b-c109e646-kserve-7d9d8884d9-b6x76\" (UID: \"53af90b8-85ec-4fa4-9831-59e5e6ce0037\") " pod="e2e-models-e2e-worker-w4-2901ec/e2e-conflict-b-c109e646-kserve-7d9d8884d9-b6x76"
Sep 14 10:02:34.890481 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:34.890442    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53af90b8-85ec-4fa4-9831-59e5e6ce0037-home\") pod \"e2e-conflict-b-c109e646-kserve-7d9d8884d9-b6x76\" (UID: \"53af90b8-85ec-4fa4-9831-59e5e6ce0037\") " pod="e2e-models-e2e-worker-w4-2901ec/e2e-conflict-b-c109e646-kserve-7d9d8884d9-b6x76"
Sep 14 10:02:34.890481 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:34.890471    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53af90b8-85ec-4fa4-9831-59e5e6ce0037-tmp-dir\") pod \"e2e-conflict-b-c109e646-kserve-7d9d8884d9-b6x76\" (UID: \"53af90b8-85ec-4fa4-9831-59e5e6ce0037\") " pod="e2e-models-e2e-worker-w4-2901ec/e2e-conflict-b-c109e646-kserve-7d9d8884d9-b6x76"
Sep 14 10:02:34.890633 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:34.890613    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53af90b8-85ec-4fa4-9831-59e5e6ce0037-model-cache\") pod \"e2e-conflict-b-c109e646-kserve-7d9d8884d9-b6x76\" (UID: \"53af90b8-85ec-4fa4-9831-59e5e6ce0037\") " pod="e2e-models-e2e-worker-w4-2901ec/e2e-conflict-b-c109e646-kserve-7d9d8884d9-b6x76"
Sep 14 10:02:34.892624 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:34.892587    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53af90b8-85ec-4fa4-9831-59e5e6ce0037-dshm\") pod \"e2e-conflict-b-c109e646-kserve-7d9d8884d9-b6x76\" (UID: \"53af90b8-85ec-4fa4-9831-59e5e6ce0037\") " pod="e2e-models-e2e-worker-w4-2901ec/e2e-conflict-b-c109e646-kserve-7d9d8884d9-b6x76"
Sep 14 10:02:34.892986 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:34.892963    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53af90b8-85ec-4fa4-9831-59e5e6ce0037-tls-certs\") pod \"e2e-conflict-b-c109e646-kserve-7d9d8884d9-b6x76\" (UID: \"53af90b8-85ec-4fa4-9831-59e5e6ce0037\") " pod="e2e-models-e2e-worker-w4-2901ec/e2e-conflict-b-c109e646-kserve-7d9d8884d9-b6x76"
Sep 14 10:02:34.898719 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:34.898683    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tb9t\" (UniqueName: \"kubernetes.io/projected/53af90b8-85ec-4fa4-9831-59e5e6ce0037-kube-api-access-8tb9t\") pod \"e2e-conflict-b-c109e646-kserve-7d9d8884d9-b6x76\" (UID: \"53af90b8-85ec-4fa4-9831-59e5e6ce0037\") " pod="e2e-models-e2e-worker-w4-2901ec/e2e-conflict-b-c109e646-kserve-7d9d8884d9-b6x76"
Sep 14 10:02:35.069596 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:35.069484    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2901ec/e2e-conflict-b-c109e646-kserve-7d9d8884d9-b6x76"
Sep 14 10:02:35.217349 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:35.217312    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-31f3b6-9k6ng" event={"ID":"b670cf8e-480f-4f68-8b9f-50b2abb9dcea","Type":"ContainerDied","Data":"64b30ebeada7107bf15d24594490f3427afaf7c0a4770af316b0b6098c7bc61e"}
Sep 14 10:02:35.217500 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:35.217378    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64b30ebeada7107bf15d24594490f3427afaf7c0a4770af316b0b6098c7bc61e"
Sep 14 10:02:35.217500 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:35.217328    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-31f3b6-9k6ng"
Sep 14 10:02:35.322742 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:35.322639    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:02:35.348328 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:35.348287    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-2901ec/e2e-conflict-b-c109e646-kserve-7d9d8884d9-b6x76"]
Sep 14 10:02:35.349297 ip-10-0-132-4 kubenswrapper[2563]: W0914 10:02:35.349267    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod53af90b8_85ec_4fa4_9831_59e5e6ce0037.slice/crio-f654e1a8e4b95fe7495f1016ae0c1907854b13a3d22045ac78f3023c2155e839 WatchSource:0}: Error finding container f654e1a8e4b95fe7495f1016ae0c1907854b13a3d22045ac78f3023c2155e839: Status 404 returned error can't find the container with id f654e1a8e4b95fe7495f1016ae0c1907854b13a3d22045ac78f3023c2155e839
Sep 14 10:02:36.209775 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:36.209736    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:02:36.226885 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:36.226848    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2901ec/e2e-conflict-b-c109e646-kserve-7d9d8884d9-b6x76" event={"ID":"53af90b8-85ec-4fa4-9831-59e5e6ce0037","Type":"ContainerStarted","Data":"e0c4f16fd7e7d45394a872be9a23d0e081f2f315507597067f5787f0a45285c3"}
Sep 14 10:02:36.227097 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:36.226892    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2901ec/e2e-conflict-b-c109e646-kserve-7d9d8884d9-b6x76" event={"ID":"53af90b8-85ec-4fa4-9831-59e5e6ce0037","Type":"ContainerStarted","Data":"f654e1a8e4b95fe7495f1016ae0c1907854b13a3d22045ac78f3023c2155e839"}
Sep 14 10:02:36.247069 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:36.246993    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-2901ec/e2e-conflict-b-c109e646-kserve-7d9d8884d9-b6x76" podStartSLOduration=2.246972577 podStartE2EDuration="2.246972577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:02:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:02:36.243599576 +0000 UTC m=+1145.283274220" watchObservedRunningTime="2026-09-14 10:02:36.246972577 +0000 UTC m=+1145.286647198"
Sep 14 10:02:37.095640 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:37.095597    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:02:37.886347 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:37.886309    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:02:38.550956 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:38.550918    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:02:38.638081 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:38.638048    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:02:38.706843 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:38.706812    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:02:38.800567 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:38.800531    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bc008e9d-6gggx"]
Sep 14 10:02:38.802106 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:38.802020    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:02:38.803563 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:38.803540    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bc008e9d-6gggx"]
Sep 14 10:02:38.884794 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:38.884748    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:02:39.135494 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:39.135401    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:02:39.236251 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:39.236212    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:02:39.483676 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:39.483587    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="148f5643-7d73-41b4-8f95-186b86ea68f2" path="/var/lib/kubelet/pods/148f5643-7d73-41b4-8f95-186b86ea68f2/volumes"
Sep 14 10:02:39.970593 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:39.970563    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:02:40.804247 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:40.804208    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:02:41.699914 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:41.699880    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:02:42.391672 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:42.391633    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:02:42.530877 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:42.530839    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:02:42.638856 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:42.638817    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:02:42.661662 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:42.661565    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-31f3b6-9k6ng"]
Sep 14 10:02:42.672183 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:42.672136    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-31f3b6-9k6ng"]
Sep 14 10:02:42.722899 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:42.722849    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:02:42.845195 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:42.845165    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:02:42.989571 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:42.989540    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:02:43.483022 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:43.482918    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b670cf8e-480f-4f68-8b9f-50b2abb9dcea" path="/var/lib/kubelet/pods/b670cf8e-480f-4f68-8b9f-50b2abb9dcea/volumes"
Sep 14 10:02:44.010860 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:44.010816    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:02:44.810369 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:44.810330    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:02:45.070472 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:45.070383    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2901ec/e2e-conflict-b-c109e646-kserve-7d9d8884d9-b6x76"
Sep 14 10:02:45.070472 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:45.070444    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-2901ec/e2e-conflict-b-c109e646-kserve-7d9d8884d9-b6x76"
Sep 14 10:02:45.084826 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:45.084791    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-2901ec/e2e-conflict-b-c109e646-kserve-7d9d8884d9-b6x76"
Sep 14 10:02:45.306637 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:45.306604    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2901ec/e2e-conflict-b-c109e646-kserve-7d9d8884d9-b6x76"
Sep 14 10:02:45.723862 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:45.723823    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:02:45.823790 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:45.823755    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:02:45.960900 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:45.960857    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f"]
Sep 14 10:02:45.961168 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:45.961122    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f" podUID="9470ce1f-5f66-492b-ac4e-a64e41e787f1" containerName="istio-proxy" containerID="cri-o://912edec05413cc804221688cf327c78beb3feb5c956c643a3b7c28b7892a7a20" gracePeriod=30
Sep 14 10:02:46.258929 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:46.258897    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:02:46.265482 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:46.265449    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-31f3b6/explicit-model-31f3b6-kserve-59cc9f967f-4sfx5" podUID="85054bf7-c54c-40aa-9693-8360bfd9b5cb" containerName="main" containerID="cri-o://21dc3730d914e5003becddbf45e2ff06baebb8cc0dc78d92364e2815dee5a587" gracePeriod=45
Sep 14 10:02:46.358460 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:46.358424    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:02:46.522843 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:46.522744    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:02:46.524229 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:46.524203    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-31f3b6/explicit-model-31f3b6-kserve-59cc9f967f-4sfx5"
Sep 14 10:02:46.625846 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:46.625798    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2k9m8\" (UniqueName: \"kubernetes.io/projected/85054bf7-c54c-40aa-9693-8360bfd9b5cb-kube-api-access-2k9m8\") pod \"85054bf7-c54c-40aa-9693-8360bfd9b5cb\" (UID: \"85054bf7-c54c-40aa-9693-8360bfd9b5cb\") "
Sep 14 10:02:46.626093 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:46.625907    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85054bf7-c54c-40aa-9693-8360bfd9b5cb-tls-certs\") pod \"85054bf7-c54c-40aa-9693-8360bfd9b5cb\" (UID: \"85054bf7-c54c-40aa-9693-8360bfd9b5cb\") "
Sep 14 10:02:46.626093 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:46.625957    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85054bf7-c54c-40aa-9693-8360bfd9b5cb-dshm\") pod \"85054bf7-c54c-40aa-9693-8360bfd9b5cb\" (UID: \"85054bf7-c54c-40aa-9693-8360bfd9b5cb\") "
Sep 14 10:02:46.626093 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:46.625976    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85054bf7-c54c-40aa-9693-8360bfd9b5cb-home\") pod \"85054bf7-c54c-40aa-9693-8360bfd9b5cb\" (UID: \"85054bf7-c54c-40aa-9693-8360bfd9b5cb\") "
Sep 14 10:02:46.626093 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:46.626046    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85054bf7-c54c-40aa-9693-8360bfd9b5cb-tmp-dir\") pod \"85054bf7-c54c-40aa-9693-8360bfd9b5cb\" (UID: \"85054bf7-c54c-40aa-9693-8360bfd9b5cb\") "
Sep 14 10:02:46.626093 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:46.626076    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85054bf7-c54c-40aa-9693-8360bfd9b5cb-model-cache\") pod \"85054bf7-c54c-40aa-9693-8360bfd9b5cb\" (UID: \"85054bf7-c54c-40aa-9693-8360bfd9b5cb\") "
Sep 14 10:02:46.626382 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:46.626306    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85054bf7-c54c-40aa-9693-8360bfd9b5cb-home" (OuterVolumeSpecName: "home") pod "85054bf7-c54c-40aa-9693-8360bfd9b5cb" (UID: "85054bf7-c54c-40aa-9693-8360bfd9b5cb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:02:46.626382 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:46.626318    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85054bf7-c54c-40aa-9693-8360bfd9b5cb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "85054bf7-c54c-40aa-9693-8360bfd9b5cb" (UID: "85054bf7-c54c-40aa-9693-8360bfd9b5cb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:02:46.626495 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:46.626441    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85054bf7-c54c-40aa-9693-8360bfd9b5cb-model-cache" (OuterVolumeSpecName: "model-cache") pod "85054bf7-c54c-40aa-9693-8360bfd9b5cb" (UID: "85054bf7-c54c-40aa-9693-8360bfd9b5cb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:02:46.626554 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:46.626541    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85054bf7-c54c-40aa-9693-8360bfd9b5cb-home\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:02:46.626605 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:46.626558    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85054bf7-c54c-40aa-9693-8360bfd9b5cb-tmp-dir\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:02:46.628287 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:46.628254    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85054bf7-c54c-40aa-9693-8360bfd9b5cb-dshm" (OuterVolumeSpecName: "dshm") pod "85054bf7-c54c-40aa-9693-8360bfd9b5cb" (UID: "85054bf7-c54c-40aa-9693-8360bfd9b5cb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:02:46.628435 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:46.628298    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85054bf7-c54c-40aa-9693-8360bfd9b5cb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "85054bf7-c54c-40aa-9693-8360bfd9b5cb" (UID: "85054bf7-c54c-40aa-9693-8360bfd9b5cb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:02:46.628435 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:46.628378    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85054bf7-c54c-40aa-9693-8360bfd9b5cb-kube-api-access-2k9m8" (OuterVolumeSpecName: "kube-api-access-2k9m8") pod "85054bf7-c54c-40aa-9693-8360bfd9b5cb" (UID: "85054bf7-c54c-40aa-9693-8360bfd9b5cb"). InnerVolumeSpecName "kube-api-access-2k9m8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:02:46.727878 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:46.727830    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k9m8\" (UniqueName: \"kubernetes.io/projected/85054bf7-c54c-40aa-9693-8360bfd9b5cb-kube-api-access-2k9m8\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:02:46.727878 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:46.727869    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85054bf7-c54c-40aa-9693-8360bfd9b5cb-tls-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:02:46.727878 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:46.727879    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85054bf7-c54c-40aa-9693-8360bfd9b5cb-dshm\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:02:46.727878 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:46.727889    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85054bf7-c54c-40aa-9693-8360bfd9b5cb-model-cache\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:02:46.864614 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:46.864523    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:02:47.310668 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:47.310628    2563 generic.go:356] "Generic (PLEG): container finished" podID="85054bf7-c54c-40aa-9693-8360bfd9b5cb" containerID="21dc3730d914e5003becddbf45e2ff06baebb8cc0dc78d92364e2815dee5a587" exitCode=0
Sep 14 10:02:47.311242 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:47.310690    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-31f3b6/explicit-model-31f3b6-kserve-59cc9f967f-4sfx5"
Sep 14 10:02:47.311242 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:47.310717    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-31f3b6/explicit-model-31f3b6-kserve-59cc9f967f-4sfx5" event={"ID":"85054bf7-c54c-40aa-9693-8360bfd9b5cb","Type":"ContainerDied","Data":"21dc3730d914e5003becddbf45e2ff06baebb8cc0dc78d92364e2815dee5a587"}
Sep 14 10:02:47.311242 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:47.310781    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-31f3b6/explicit-model-31f3b6-kserve-59cc9f967f-4sfx5" event={"ID":"85054bf7-c54c-40aa-9693-8360bfd9b5cb","Type":"ContainerDied","Data":"c322d97bb133cbba8897dc88933fa895642fdd97b0bed149fa9a761ce9755d11"}
Sep 14 10:02:47.311242 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:47.310807    2563 scope.go:117] "RemoveContainer" containerID="21dc3730d914e5003becddbf45e2ff06baebb8cc0dc78d92364e2815dee5a587"
Sep 14 10:02:47.323049 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:47.323028    2563 scope.go:117] "RemoveContainer" containerID="21dc3730d914e5003becddbf45e2ff06baebb8cc0dc78d92364e2815dee5a587"
Sep 14 10:02:47.323421 ip-10-0-132-4 kubenswrapper[2563]: E0914 10:02:47.323400    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21dc3730d914e5003becddbf45e2ff06baebb8cc0dc78d92364e2815dee5a587\": container with ID starting with 21dc3730d914e5003becddbf45e2ff06baebb8cc0dc78d92364e2815dee5a587 not found: ID does not exist" containerID="21dc3730d914e5003becddbf45e2ff06baebb8cc0dc78d92364e2815dee5a587"
Sep 14 10:02:47.323474 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:47.323433    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21dc3730d914e5003becddbf45e2ff06baebb8cc0dc78d92364e2815dee5a587"} err="failed to get container status \"21dc3730d914e5003becddbf45e2ff06baebb8cc0dc78d92364e2815dee5a587\": rpc error: code = NotFound desc = could not find container \"21dc3730d914e5003becddbf45e2ff06baebb8cc0dc78d92364e2815dee5a587\": container with ID starting with 21dc3730d914e5003becddbf45e2ff06baebb8cc0dc78d92364e2815dee5a587 not found: ID does not exist"
Sep 14 10:02:47.335904 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:47.335864    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-31f3b6/explicit-model-31f3b6-kserve-59cc9f967f-4sfx5"]
Sep 14 10:02:47.341877 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:47.341841    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-31f3b6/explicit-model-31f3b6-kserve-59cc9f967f-4sfx5"]
Sep 14 10:02:47.482845 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:47.482806    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85054bf7-c54c-40aa-9693-8360bfd9b5cb" path="/var/lib/kubelet/pods/85054bf7-c54c-40aa-9693-8360bfd9b5cb/volumes"
Sep 14 10:02:47.800323 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:47.800275    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:02:48.832249 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:48.832210    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:02:49.467502 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:49.467436    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:02:49.595459 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:49.595417    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:02:49.783437 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:49.783393    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48"]
Sep 14 10:02:49.783645 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:49.783617    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48" podUID="6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d" containerName="istio-proxy" containerID="cri-o://f0a0884c1395aee0f96026b91447e3cb861a911d6e9aa38bb13494e12697dc1d" gracePeriod=30
Sep 14 10:02:49.796305 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:49.796268    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:02:49.916235 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:49.916193    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:02:50.217415 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:50.217318    2563 patch_prober.go:28] interesting pod/e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 14 10:02:50.217415 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:50.217399    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48" podUID="6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 14 10:02:50.244145 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:50.244112    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:02:50.600593 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:50.600557    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:02:50.990426 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:50.990389    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:02:51.229354 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:51.229270    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f"
Sep 14 10:02:51.343129 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:51.343095    2563 generic.go:356] "Generic (PLEG): container finished" podID="9470ce1f-5f66-492b-ac4e-a64e41e787f1" containerID="912edec05413cc804221688cf327c78beb3feb5c956c643a3b7c28b7892a7a20" exitCode=0
Sep 14 10:02:51.343359 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:51.343165    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f"
Sep 14 10:02:51.343359 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:51.343184    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f" event={"ID":"9470ce1f-5f66-492b-ac4e-a64e41e787f1","Type":"ContainerDied","Data":"912edec05413cc804221688cf327c78beb3feb5c956c643a3b7c28b7892a7a20"}
Sep 14 10:02:51.343359 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:51.343249    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f" event={"ID":"9470ce1f-5f66-492b-ac4e-a64e41e787f1","Type":"ContainerDied","Data":"7e0700a914edda6956bda38e8af5d6f7f8c02f5903f37cd490ae31bfd1a23d09"}
Sep 14 10:02:51.343359 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:51.343286    2563 scope.go:117] "RemoveContainer" containerID="912edec05413cc804221688cf327c78beb3feb5c956c643a3b7c28b7892a7a20"
Sep 14 10:02:51.353562 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:51.353534    2563 scope.go:117] "RemoveContainer" containerID="2fa2e9c50435513b7fa25e375bfca295b74eee1a87b076cb6478e50a2c3db991"
Sep 14 10:02:51.363921 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:51.363897    2563 scope.go:117] "RemoveContainer" containerID="912edec05413cc804221688cf327c78beb3feb5c956c643a3b7c28b7892a7a20"
Sep 14 10:02:51.364332 ip-10-0-132-4 kubenswrapper[2563]: E0914 10:02:51.364304    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"912edec05413cc804221688cf327c78beb3feb5c956c643a3b7c28b7892a7a20\": container with ID starting with 912edec05413cc804221688cf327c78beb3feb5c956c643a3b7c28b7892a7a20 not found: ID does not exist" containerID="912edec05413cc804221688cf327c78beb3feb5c956c643a3b7c28b7892a7a20"
Sep 14 10:02:51.364436 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:51.364341    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"912edec05413cc804221688cf327c78beb3feb5c956c643a3b7c28b7892a7a20"} err="failed to get container status \"912edec05413cc804221688cf327c78beb3feb5c956c643a3b7c28b7892a7a20\": rpc error: code = NotFound desc = could not find container \"912edec05413cc804221688cf327c78beb3feb5c956c643a3b7c28b7892a7a20\": container with ID starting with 912edec05413cc804221688cf327c78beb3feb5c956c643a3b7c28b7892a7a20 not found: ID does not exist"
Sep 14 10:02:51.364436 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:51.364361    2563 scope.go:117] "RemoveContainer" containerID="2fa2e9c50435513b7fa25e375bfca295b74eee1a87b076cb6478e50a2c3db991"
Sep 14 10:02:51.364647 ip-10-0-132-4 kubenswrapper[2563]: E0914 10:02:51.364628    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fa2e9c50435513b7fa25e375bfca295b74eee1a87b076cb6478e50a2c3db991\": container with ID starting with 2fa2e9c50435513b7fa25e375bfca295b74eee1a87b076cb6478e50a2c3db991 not found: ID does not exist" containerID="2fa2e9c50435513b7fa25e375bfca295b74eee1a87b076cb6478e50a2c3db991"
Sep 14 10:02:51.364688 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:51.364654    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fa2e9c50435513b7fa25e375bfca295b74eee1a87b076cb6478e50a2c3db991"} err="failed to get container status \"2fa2e9c50435513b7fa25e375bfca295b74eee1a87b076cb6478e50a2c3db991\": rpc error: code = NotFound desc = could not find container \"2fa2e9c50435513b7fa25e375bfca295b74eee1a87b076cb6478e50a2c3db991\": container with ID starting with 2fa2e9c50435513b7fa25e375bfca295b74eee1a87b076cb6478e50a2c3db991 not found: ID does not exist"
Sep 14 10:02:51.382162 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:51.382112    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9470ce1f-5f66-492b-ac4e-a64e41e787f1-istiod-ca-cert\") pod \"9470ce1f-5f66-492b-ac4e-a64e41e787f1\" (UID: \"9470ce1f-5f66-492b-ac4e-a64e41e787f1\") "
Sep 14 10:02:51.382329 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:51.382201    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9470ce1f-5f66-492b-ac4e-a64e41e787f1-istio-data\") pod \"9470ce1f-5f66-492b-ac4e-a64e41e787f1\" (UID: \"9470ce1f-5f66-492b-ac4e-a64e41e787f1\") "
Sep 14 10:02:51.382329 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:51.382226    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9470ce1f-5f66-492b-ac4e-a64e41e787f1-istio-podinfo\") pod \"9470ce1f-5f66-492b-ac4e-a64e41e787f1\" (UID: \"9470ce1f-5f66-492b-ac4e-a64e41e787f1\") "
Sep 14 10:02:51.382329 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:51.382274    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9470ce1f-5f66-492b-ac4e-a64e41e787f1-istio-envoy\") pod \"9470ce1f-5f66-492b-ac4e-a64e41e787f1\" (UID: \"9470ce1f-5f66-492b-ac4e-a64e41e787f1\") "
Sep 14 10:02:51.382329 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:51.382303    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hld8q\" (UniqueName: \"kubernetes.io/projected/9470ce1f-5f66-492b-ac4e-a64e41e787f1-kube-api-access-hld8q\") pod \"9470ce1f-5f66-492b-ac4e-a64e41e787f1\" (UID: \"9470ce1f-5f66-492b-ac4e-a64e41e787f1\") "
Sep 14 10:02:51.382505 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:51.382365    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9470ce1f-5f66-492b-ac4e-a64e41e787f1-credential-socket\") pod \"9470ce1f-5f66-492b-ac4e-a64e41e787f1\" (UID: \"9470ce1f-5f66-492b-ac4e-a64e41e787f1\") "
Sep 14 10:02:51.382505 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:51.382425    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9470ce1f-5f66-492b-ac4e-a64e41e787f1-workload-socket\") pod \"9470ce1f-5f66-492b-ac4e-a64e41e787f1\" (UID: \"9470ce1f-5f66-492b-ac4e-a64e41e787f1\") "
Sep 14 10:02:51.382505 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:51.382451    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9470ce1f-5f66-492b-ac4e-a64e41e787f1-istio-token\") pod \"9470ce1f-5f66-492b-ac4e-a64e41e787f1\" (UID: \"9470ce1f-5f66-492b-ac4e-a64e41e787f1\") "
Sep 14 10:02:51.382505 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:51.382487    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9470ce1f-5f66-492b-ac4e-a64e41e787f1-workload-certs\") pod \"9470ce1f-5f66-492b-ac4e-a64e41e787f1\" (UID: \"9470ce1f-5f66-492b-ac4e-a64e41e787f1\") "
Sep 14 10:02:51.382706 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:51.382593    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9470ce1f-5f66-492b-ac4e-a64e41e787f1-istio-data" (OuterVolumeSpecName: "istio-data") pod "9470ce1f-5f66-492b-ac4e-a64e41e787f1" (UID: "9470ce1f-5f66-492b-ac4e-a64e41e787f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:02:51.382706 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:51.382639    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9470ce1f-5f66-492b-ac4e-a64e41e787f1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9470ce1f-5f66-492b-ac4e-a64e41e787f1" (UID: "9470ce1f-5f66-492b-ac4e-a64e41e787f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:02:51.382894 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:51.382750    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9470ce1f-5f66-492b-ac4e-a64e41e787f1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9470ce1f-5f66-492b-ac4e-a64e41e787f1" (UID: "9470ce1f-5f66-492b-ac4e-a64e41e787f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:02:51.382894 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:51.382759    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9470ce1f-5f66-492b-ac4e-a64e41e787f1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9470ce1f-5f66-492b-ac4e-a64e41e787f1" (UID: "9470ce1f-5f66-492b-ac4e-a64e41e787f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:02:51.382894 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:51.382886    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9470ce1f-5f66-492b-ac4e-a64e41e787f1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9470ce1f-5f66-492b-ac4e-a64e41e787f1" (UID: "9470ce1f-5f66-492b-ac4e-a64e41e787f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:02:51.383093 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:51.383024    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9470ce1f-5f66-492b-ac4e-a64e41e787f1-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:02:51.383093 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:51.383039    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9470ce1f-5f66-492b-ac4e-a64e41e787f1-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:02:51.383093 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:51.383052    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9470ce1f-5f66-492b-ac4e-a64e41e787f1-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:02:51.383093 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:51.383068    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9470ce1f-5f66-492b-ac4e-a64e41e787f1-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:02:51.383093 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:51.383085    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9470ce1f-5f66-492b-ac4e-a64e41e787f1-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:02:51.384735 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:51.384701    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9470ce1f-5f66-492b-ac4e-a64e41e787f1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9470ce1f-5f66-492b-ac4e-a64e41e787f1" (UID: "9470ce1f-5f66-492b-ac4e-a64e41e787f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:02:51.384887 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:51.384779    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9470ce1f-5f66-492b-ac4e-a64e41e787f1-kube-api-access-hld8q" (OuterVolumeSpecName: "kube-api-access-hld8q") pod "9470ce1f-5f66-492b-ac4e-a64e41e787f1" (UID: "9470ce1f-5f66-492b-ac4e-a64e41e787f1"). InnerVolumeSpecName "kube-api-access-hld8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:02:51.384887 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:51.384787    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9470ce1f-5f66-492b-ac4e-a64e41e787f1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9470ce1f-5f66-492b-ac4e-a64e41e787f1" (UID: "9470ce1f-5f66-492b-ac4e-a64e41e787f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 10:02:51.384887 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:51.384814    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9470ce1f-5f66-492b-ac4e-a64e41e787f1-istio-token" (OuterVolumeSpecName: "istio-token") pod "9470ce1f-5f66-492b-ac4e-a64e41e787f1" (UID: "9470ce1f-5f66-492b-ac4e-a64e41e787f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:02:51.483893 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:51.483803    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9470ce1f-5f66-492b-ac4e-a64e41e787f1-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:02:51.483893 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:51.483831    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9470ce1f-5f66-492b-ac4e-a64e41e787f1-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:02:51.483893 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:51.483840    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9470ce1f-5f66-492b-ac4e-a64e41e787f1-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:02:51.483893 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:51.483848    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hld8q\" (UniqueName: \"kubernetes.io/projected/9470ce1f-5f66-492b-ac4e-a64e41e787f1-kube-api-access-hld8q\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:02:51.663058 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:51.663017    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f"]
Sep 14 10:02:51.666612 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:51.666567    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-bc008e9d-openshift-default-5876b44bd8-mq66f"]
Sep 14 10:02:51.760490 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:51.760449    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:02:51.849610 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:51.849576    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:02:51.950466 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:51.950431    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:02:52.827340 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:52.827301    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:02:53.484336 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:53.484249    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9470ce1f-5f66-492b-ac4e-a64e41e787f1" path="/var/lib/kubelet/pods/9470ce1f-5f66-492b-ac4e-a64e41e787f1/volumes"
Sep 14 10:02:54.049787 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:54.049752    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:02:54.219177 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:54.219131    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:02:54.324951 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:54.324865    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:02:54.421617 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:54.421586    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:02:55.040589 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:55.040561    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48"
Sep 14 10:02:55.121326 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:55.121219    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d-istiod-ca-cert\") pod \"6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d\" (UID: \"6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d\") "
Sep 14 10:02:55.121326 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:55.121272    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d-workload-certs\") pod \"6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d\" (UID: \"6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d\") "
Sep 14 10:02:55.121326 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:55.121320    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d-istio-podinfo\") pod \"6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d\" (UID: \"6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d\") "
Sep 14 10:02:55.121939 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:55.121420    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j4jjr\" (UniqueName: \"kubernetes.io/projected/6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d-kube-api-access-j4jjr\") pod \"6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d\" (UID: \"6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d\") "
Sep 14 10:02:55.121939 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:55.121456    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d-workload-socket\") pod \"6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d\" (UID: \"6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d\") "
Sep 14 10:02:55.121939 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:55.121488    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d-istio-envoy\") pod \"6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d\" (UID: \"6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d\") "
Sep 14 10:02:55.121939 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:55.121525    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d-credential-socket\") pod \"6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d\" (UID: \"6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d\") "
Sep 14 10:02:55.121939 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:55.121595    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d" (UID: "6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:02:55.121939 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:55.121643    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d-istio-data\") pod \"6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d\" (UID: \"6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d\") "
Sep 14 10:02:55.121939 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:55.121705    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d-istio-token\") pod \"6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d\" (UID: \"6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d\") "
Sep 14 10:02:55.121939 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:55.121718    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d" (UID: "6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:02:55.121939 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:55.121752    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d" (UID: "6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:02:55.121939 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:55.121886    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d-istio-data" (OuterVolumeSpecName: "istio-data") pod "6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d" (UID: "6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:02:55.122431 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:55.122082    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d" (UID: "6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:02:55.122431 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:55.122202    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:02:55.122431 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:55.122234    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:02:55.122431 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:55.122248    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:02:55.122431 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:55.122264    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:02:55.123805 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:55.123770    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d" (UID: "6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:02:55.123912 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:55.123873    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d-kube-api-access-j4jjr" (OuterVolumeSpecName: "kube-api-access-j4jjr") pod "6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d" (UID: "6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d"). InnerVolumeSpecName "kube-api-access-j4jjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:02:55.124041 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:55.124020    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d" (UID: "6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 10:02:55.124171 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:55.124154    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d-istio-token" (OuterVolumeSpecName: "istio-token") pod "6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d" (UID: "6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:02:55.222859 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:55.222817    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:02:55.222859 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:55.222852    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4jjr\" (UniqueName: \"kubernetes.io/projected/6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d-kube-api-access-j4jjr\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:02:55.222859 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:55.222861    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:02:55.222859 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:55.222871    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:02:55.223183 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:55.222879    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:02:55.372544 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:55.372444    2563 generic.go:356] "Generic (PLEG): container finished" podID="6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d" containerID="f0a0884c1395aee0f96026b91447e3cb861a911d6e9aa38bb13494e12697dc1d" exitCode=0
Sep 14 10:02:55.372544 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:55.372500    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48"
Sep 14 10:02:55.372748 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:55.372553    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48" event={"ID":"6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d","Type":"ContainerDied","Data":"f0a0884c1395aee0f96026b91447e3cb861a911d6e9aa38bb13494e12697dc1d"}
Sep 14 10:02:55.372748 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:55.372626    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48" event={"ID":"6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d","Type":"ContainerDied","Data":"171abaaaca437829182c694ed964ecb747c77f50d098daba1f3f96e50fa48493"}
Sep 14 10:02:55.372748 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:55.372652    2563 scope.go:117] "RemoveContainer" containerID="f0a0884c1395aee0f96026b91447e3cb861a911d6e9aa38bb13494e12697dc1d"
Sep 14 10:02:55.383871 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:55.383844    2563 scope.go:117] "RemoveContainer" containerID="a73865c4e4cf5831364578f49708bffeb619db64eecfcae81eb51b77470e959c"
Sep 14 10:02:55.394038 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:55.393986    2563 scope.go:117] "RemoveContainer" containerID="f0a0884c1395aee0f96026b91447e3cb861a911d6e9aa38bb13494e12697dc1d"
Sep 14 10:02:55.394421 ip-10-0-132-4 kubenswrapper[2563]: E0914 10:02:55.394400    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0a0884c1395aee0f96026b91447e3cb861a911d6e9aa38bb13494e12697dc1d\": container with ID starting with f0a0884c1395aee0f96026b91447e3cb861a911d6e9aa38bb13494e12697dc1d not found: ID does not exist" containerID="f0a0884c1395aee0f96026b91447e3cb861a911d6e9aa38bb13494e12697dc1d"
Sep 14 10:02:55.394493 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:55.394431    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0a0884c1395aee0f96026b91447e3cb861a911d6e9aa38bb13494e12697dc1d"} err="failed to get container status \"f0a0884c1395aee0f96026b91447e3cb861a911d6e9aa38bb13494e12697dc1d\": rpc error: code = NotFound desc = could not find container \"f0a0884c1395aee0f96026b91447e3cb861a911d6e9aa38bb13494e12697dc1d\": container with ID starting with f0a0884c1395aee0f96026b91447e3cb861a911d6e9aa38bb13494e12697dc1d not found: ID does not exist"
Sep 14 10:02:55.394493 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:55.394450    2563 scope.go:117] "RemoveContainer" containerID="a73865c4e4cf5831364578f49708bffeb619db64eecfcae81eb51b77470e959c"
Sep 14 10:02:55.394724 ip-10-0-132-4 kubenswrapper[2563]: E0914 10:02:55.394706    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a73865c4e4cf5831364578f49708bffeb619db64eecfcae81eb51b77470e959c\": container with ID starting with a73865c4e4cf5831364578f49708bffeb619db64eecfcae81eb51b77470e959c not found: ID does not exist" containerID="a73865c4e4cf5831364578f49708bffeb619db64eecfcae81eb51b77470e959c"
Sep 14 10:02:55.394767 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:55.394726    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a73865c4e4cf5831364578f49708bffeb619db64eecfcae81eb51b77470e959c"} err="failed to get container status \"a73865c4e4cf5831364578f49708bffeb619db64eecfcae81eb51b77470e959c\": rpc error: code = NotFound desc = could not find container \"a73865c4e4cf5831364578f49708bffeb619db64eecfcae81eb51b77470e959c\": container with ID starting with a73865c4e4cf5831364578f49708bffeb619db64eecfcae81eb51b77470e959c not found: ID does not exist"
Sep 14 10:02:55.399951 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:55.399910    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48"]
Sep 14 10:02:55.404784 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:55.404739    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-31f3b6-openshift-default-dddc5cf6-57w48"]
Sep 14 10:02:55.483367 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:02:55.483327    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d" path="/var/lib/kubelet/pods/6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d/volumes"
Sep 14 10:03:00.608186 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:00.608153    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:03:00.706914 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:00.706878    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:03:01.555124 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:01.555088    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:03:01.789336 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:01.789295    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:03:02.057410 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:02.057374    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:03:03.778637 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:03.778600    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:03:04.430706 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:04.430665    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:03:04.525628 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:04.525594    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:03:04.694931 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:04.694829    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:03:05.780585 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:05.780543    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:03:06.554284 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:06.554244    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-b6812e/distinct-w0-b6812e-kserve-786cffbfd5-q74np"]
Sep 14 10:03:06.554649 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:06.554636    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85054bf7-c54c-40aa-9693-8360bfd9b5cb" containerName="main"
Sep 14 10:03:06.554700 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:06.554651    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="85054bf7-c54c-40aa-9693-8360bfd9b5cb" containerName="main"
Sep 14 10:03:06.554700 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:06.554672    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d" containerName="istio-proxy"
Sep 14 10:03:06.554700 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:06.554677    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d" containerName="istio-proxy"
Sep 14 10:03:06.554700 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:06.554688    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9470ce1f-5f66-492b-ac4e-a64e41e787f1" containerName="istio-proxy"
Sep 14 10:03:06.554700 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:06.554694    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9470ce1f-5f66-492b-ac4e-a64e41e787f1" containerName="istio-proxy"
Sep 14 10:03:06.554700 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:06.554700    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d" containerName="istio-proxy"
Sep 14 10:03:06.554914 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:06.554706    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d" containerName="istio-proxy"
Sep 14 10:03:06.554914 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:06.554765    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9470ce1f-5f66-492b-ac4e-a64e41e787f1" containerName="istio-proxy"
Sep 14 10:03:06.554914 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:06.554772    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="85054bf7-c54c-40aa-9693-8360bfd9b5cb" containerName="main"
Sep 14 10:03:06.554914 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:06.554780    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d" containerName="istio-proxy"
Sep 14 10:03:06.554914 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:06.554787    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ebfd43a-3338-44c7-8c43-5f42c3a5ad1d" containerName="istio-proxy"
Sep 14 10:03:06.554914 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:06.554792    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9470ce1f-5f66-492b-ac4e-a64e41e787f1" containerName="istio-proxy"
Sep 14 10:03:06.558114 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:06.558089    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b6812e/distinct-w0-b6812e-kserve-786cffbfd5-q74np"
Sep 14 10:03:06.560672 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:06.560643    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b6812e\"/\"distinct-w0-b6812e-kserve-self-signed-certs\""
Sep 14 10:03:06.569558 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:06.569524    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b6812e/distinct-w0-b6812e-kserve-786cffbfd5-q74np"]
Sep 14 10:03:06.584501 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:06.584447    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:03:06.641854 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:06.641808    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60fe835c-e5f9-44ae-9dbc-eff55a5bc50a-tmp-dir\") pod \"distinct-w0-b6812e-kserve-786cffbfd5-q74np\" (UID: \"60fe835c-e5f9-44ae-9dbc-eff55a5bc50a\") " pod="e2e-models-e2e-worker-w0-b6812e/distinct-w0-b6812e-kserve-786cffbfd5-q74np"
Sep 14 10:03:06.641854 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:06.641864    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60fe835c-e5f9-44ae-9dbc-eff55a5bc50a-tls-certs\") pod \"distinct-w0-b6812e-kserve-786cffbfd5-q74np\" (UID: \"60fe835c-e5f9-44ae-9dbc-eff55a5bc50a\") " pod="e2e-models-e2e-worker-w0-b6812e/distinct-w0-b6812e-kserve-786cffbfd5-q74np"
Sep 14 10:03:06.642131 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:06.641896    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz22f\" (UniqueName: \"kubernetes.io/projected/60fe835c-e5f9-44ae-9dbc-eff55a5bc50a-kube-api-access-rz22f\") pod \"distinct-w0-b6812e-kserve-786cffbfd5-q74np\" (UID: \"60fe835c-e5f9-44ae-9dbc-eff55a5bc50a\") " pod="e2e-models-e2e-worker-w0-b6812e/distinct-w0-b6812e-kserve-786cffbfd5-q74np"
Sep 14 10:03:06.642131 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:06.641918    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60fe835c-e5f9-44ae-9dbc-eff55a5bc50a-dshm\") pod \"distinct-w0-b6812e-kserve-786cffbfd5-q74np\" (UID: \"60fe835c-e5f9-44ae-9dbc-eff55a5bc50a\") " pod="e2e-models-e2e-worker-w0-b6812e/distinct-w0-b6812e-kserve-786cffbfd5-q74np"
Sep 14 10:03:06.642131 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:06.641968    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60fe835c-e5f9-44ae-9dbc-eff55a5bc50a-model-cache\") pod \"distinct-w0-b6812e-kserve-786cffbfd5-q74np\" (UID: \"60fe835c-e5f9-44ae-9dbc-eff55a5bc50a\") " pod="e2e-models-e2e-worker-w0-b6812e/distinct-w0-b6812e-kserve-786cffbfd5-q74np"
Sep 14 10:03:06.642131 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:06.641986    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60fe835c-e5f9-44ae-9dbc-eff55a5bc50a-home\") pod \"distinct-w0-b6812e-kserve-786cffbfd5-q74np\" (UID: \"60fe835c-e5f9-44ae-9dbc-eff55a5bc50a\") " pod="e2e-models-e2e-worker-w0-b6812e/distinct-w0-b6812e-kserve-786cffbfd5-q74np"
Sep 14 10:03:06.683674 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:06.683637    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:03:06.742770 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:06.742730    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60fe835c-e5f9-44ae-9dbc-eff55a5bc50a-model-cache\") pod \"distinct-w0-b6812e-kserve-786cffbfd5-q74np\" (UID: \"60fe835c-e5f9-44ae-9dbc-eff55a5bc50a\") " pod="e2e-models-e2e-worker-w0-b6812e/distinct-w0-b6812e-kserve-786cffbfd5-q74np"
Sep 14 10:03:06.742770 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:06.742770    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60fe835c-e5f9-44ae-9dbc-eff55a5bc50a-home\") pod \"distinct-w0-b6812e-kserve-786cffbfd5-q74np\" (UID: \"60fe835c-e5f9-44ae-9dbc-eff55a5bc50a\") " pod="e2e-models-e2e-worker-w0-b6812e/distinct-w0-b6812e-kserve-786cffbfd5-q74np"
Sep 14 10:03:06.742989 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:06.742850    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60fe835c-e5f9-44ae-9dbc-eff55a5bc50a-tmp-dir\") pod \"distinct-w0-b6812e-kserve-786cffbfd5-q74np\" (UID: \"60fe835c-e5f9-44ae-9dbc-eff55a5bc50a\") " pod="e2e-models-e2e-worker-w0-b6812e/distinct-w0-b6812e-kserve-786cffbfd5-q74np"
Sep 14 10:03:06.742989 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:06.742899    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60fe835c-e5f9-44ae-9dbc-eff55a5bc50a-tls-certs\") pod \"distinct-w0-b6812e-kserve-786cffbfd5-q74np\" (UID: \"60fe835c-e5f9-44ae-9dbc-eff55a5bc50a\") " pod="e2e-models-e2e-worker-w0-b6812e/distinct-w0-b6812e-kserve-786cffbfd5-q74np"
Sep 14 10:03:06.742989 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:06.742916    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz22f\" (UniqueName: \"kubernetes.io/projected/60fe835c-e5f9-44ae-9dbc-eff55a5bc50a-kube-api-access-rz22f\") pod \"distinct-w0-b6812e-kserve-786cffbfd5-q74np\" (UID: \"60fe835c-e5f9-44ae-9dbc-eff55a5bc50a\") " pod="e2e-models-e2e-worker-w0-b6812e/distinct-w0-b6812e-kserve-786cffbfd5-q74np"
Sep 14 10:03:06.742989 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:06.742935    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60fe835c-e5f9-44ae-9dbc-eff55a5bc50a-dshm\") pod \"distinct-w0-b6812e-kserve-786cffbfd5-q74np\" (UID: \"60fe835c-e5f9-44ae-9dbc-eff55a5bc50a\") " pod="e2e-models-e2e-worker-w0-b6812e/distinct-w0-b6812e-kserve-786cffbfd5-q74np"
Sep 14 10:03:06.743274 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:06.743250    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60fe835c-e5f9-44ae-9dbc-eff55a5bc50a-model-cache\") pod \"distinct-w0-b6812e-kserve-786cffbfd5-q74np\" (UID: \"60fe835c-e5f9-44ae-9dbc-eff55a5bc50a\") " pod="e2e-models-e2e-worker-w0-b6812e/distinct-w0-b6812e-kserve-786cffbfd5-q74np"
Sep 14 10:03:06.743329 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:06.743275    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60fe835c-e5f9-44ae-9dbc-eff55a5bc50a-home\") pod \"distinct-w0-b6812e-kserve-786cffbfd5-q74np\" (UID: \"60fe835c-e5f9-44ae-9dbc-eff55a5bc50a\") " pod="e2e-models-e2e-worker-w0-b6812e/distinct-w0-b6812e-kserve-786cffbfd5-q74np"
Sep 14 10:03:06.743329 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:06.743308    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60fe835c-e5f9-44ae-9dbc-eff55a5bc50a-tmp-dir\") pod \"distinct-w0-b6812e-kserve-786cffbfd5-q74np\" (UID: \"60fe835c-e5f9-44ae-9dbc-eff55a5bc50a\") " pod="e2e-models-e2e-worker-w0-b6812e/distinct-w0-b6812e-kserve-786cffbfd5-q74np"
Sep 14 10:03:06.745459 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:06.745430    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60fe835c-e5f9-44ae-9dbc-eff55a5bc50a-dshm\") pod \"distinct-w0-b6812e-kserve-786cffbfd5-q74np\" (UID: \"60fe835c-e5f9-44ae-9dbc-eff55a5bc50a\") " pod="e2e-models-e2e-worker-w0-b6812e/distinct-w0-b6812e-kserve-786cffbfd5-q74np"
Sep 14 10:03:06.745731 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:06.745712    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60fe835c-e5f9-44ae-9dbc-eff55a5bc50a-tls-certs\") pod \"distinct-w0-b6812e-kserve-786cffbfd5-q74np\" (UID: \"60fe835c-e5f9-44ae-9dbc-eff55a5bc50a\") " pod="e2e-models-e2e-worker-w0-b6812e/distinct-w0-b6812e-kserve-786cffbfd5-q74np"
Sep 14 10:03:06.751422 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:06.751391    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz22f\" (UniqueName: \"kubernetes.io/projected/60fe835c-e5f9-44ae-9dbc-eff55a5bc50a-kube-api-access-rz22f\") pod \"distinct-w0-b6812e-kserve-786cffbfd5-q74np\" (UID: \"60fe835c-e5f9-44ae-9dbc-eff55a5bc50a\") " pod="e2e-models-e2e-worker-w0-b6812e/distinct-w0-b6812e-kserve-786cffbfd5-q74np"
Sep 14 10:03:06.870833 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:06.870737    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b6812e/distinct-w0-b6812e-kserve-786cffbfd5-q74np"
Sep 14 10:03:07.150935 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:07.150900    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b6812e/distinct-w0-b6812e-kserve-786cffbfd5-q74np"]
Sep 14 10:03:07.153120 ip-10-0-132-4 kubenswrapper[2563]: W0914 10:03:07.153076    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod60fe835c_e5f9_44ae_9dbc_eff55a5bc50a.slice/crio-0cd2f339b16fddb2c2a35b0ec9d1e2864d2f75250145eacdb09478ef5eebfaff WatchSource:0}: Error finding container 0cd2f339b16fddb2c2a35b0ec9d1e2864d2f75250145eacdb09478ef5eebfaff: Status 404 returned error can't find the container with id 0cd2f339b16fddb2c2a35b0ec9d1e2864d2f75250145eacdb09478ef5eebfaff
Sep 14 10:03:07.457035 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:07.456917    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b6812e/distinct-w0-b6812e-kserve-786cffbfd5-q74np" event={"ID":"60fe835c-e5f9-44ae-9dbc-eff55a5bc50a","Type":"ContainerStarted","Data":"195a48f81b5af2399f25f6ef37137f2cf814c3dcb985649efd845f04e0fcded5"}
Sep 14 10:03:07.457035 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:07.456965    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b6812e/distinct-w0-b6812e-kserve-786cffbfd5-q74np" event={"ID":"60fe835c-e5f9-44ae-9dbc-eff55a5bc50a","Type":"ContainerStarted","Data":"0cd2f339b16fddb2c2a35b0ec9d1e2864d2f75250145eacdb09478ef5eebfaff"}
Sep 14 10:03:07.583614 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:07.583553    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:03:08.497416 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:08.497380    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:03:08.576407 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:08.576367    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:03:08.675643 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:08.675612    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:03:09.009584 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:09.009548    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:03:09.113316 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:09.113282    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:03:16.031197 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:16.031160    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:03:16.871502 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:16.871448    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-b6812e/distinct-w0-b6812e-kserve-786cffbfd5-q74np"
Sep 14 10:03:16.871718 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:16.871531    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b6812e/distinct-w0-b6812e-kserve-786cffbfd5-q74np"
Sep 14 10:03:16.884458 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:16.884428    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-b6812e/distinct-w0-b6812e-kserve-786cffbfd5-q74np"
Sep 14 10:03:16.905520 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:16.905459    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-b6812e/distinct-w0-b6812e-kserve-786cffbfd5-q74np" podStartSLOduration=10.905440405 podStartE2EDuration="10.905440405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:03:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:03:07.481591829 +0000 UTC m=+1176.521266454" watchObservedRunningTime="2026-09-14 10:03:16.905440405 +0000 UTC m=+1185.945115025"
Sep 14 10:03:17.542339 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:17.542306    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b6812e/distinct-w0-b6812e-kserve-786cffbfd5-q74np"
Sep 14 10:03:29.662180 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:29.662136    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:03:31.280322 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:31.280289    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:03:32.168063 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:32.168020    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:03:32.968769 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:32.968733    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:03:34.001783 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:34.001750    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:03:34.430114 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:34.429996    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:03:34.761149 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:34.761118    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:03:34.861277 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:34.861245    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:03:35.042603 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:35.042504    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7"]
Sep 14 10:03:35.043027 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:35.042932    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9470ce1f-5f66-492b-ac4e-a64e41e787f1" containerName="istio-proxy"
Sep 14 10:03:35.043027 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:35.042944    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9470ce1f-5f66-492b-ac4e-a64e41e787f1" containerName="istio-proxy"
Sep 14 10:03:35.052682 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:35.052648    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7"
Sep 14 10:03:35.071842 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:35.071811    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-9e19b7-openshift-default-dockercfg-62gvr\""
Sep 14 10:03:35.089622 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:35.089584    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7"]
Sep 14 10:03:35.107487 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:35.107443    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:03:35.247066 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:35.246978    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beb8592b-6036-475a-b404-6acdf015fab2-workload-certs\") pod \"e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7\" (UID: \"beb8592b-6036-475a-b404-6acdf015fab2\") " pod="openshift-ingress/e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7"
Sep 14 10:03:35.247313 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:35.247102    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beb8592b-6036-475a-b404-6acdf015fab2-workload-socket\") pod \"e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7\" (UID: \"beb8592b-6036-475a-b404-6acdf015fab2\") " pod="openshift-ingress/e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7"
Sep 14 10:03:35.247313 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:35.247160    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beb8592b-6036-475a-b404-6acdf015fab2-credential-socket\") pod \"e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7\" (UID: \"beb8592b-6036-475a-b404-6acdf015fab2\") " pod="openshift-ingress/e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7"
Sep 14 10:03:35.247313 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:35.247198    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beb8592b-6036-475a-b404-6acdf015fab2-istio-podinfo\") pod \"e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7\" (UID: \"beb8592b-6036-475a-b404-6acdf015fab2\") " pod="openshift-ingress/e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7"
Sep 14 10:03:35.247313 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:35.247229    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beb8592b-6036-475a-b404-6acdf015fab2-istio-token\") pod \"e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7\" (UID: \"beb8592b-6036-475a-b404-6acdf015fab2\") " pod="openshift-ingress/e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7"
Sep 14 10:03:35.247313 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:35.247275    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beb8592b-6036-475a-b404-6acdf015fab2-istio-envoy\") pod \"e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7\" (UID: \"beb8592b-6036-475a-b404-6acdf015fab2\") " pod="openshift-ingress/e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7"
Sep 14 10:03:35.247497 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:35.247334    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq4zj\" (UniqueName: \"kubernetes.io/projected/beb8592b-6036-475a-b404-6acdf015fab2-kube-api-access-bq4zj\") pod \"e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7\" (UID: \"beb8592b-6036-475a-b404-6acdf015fab2\") " pod="openshift-ingress/e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7"
Sep 14 10:03:35.247497 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:35.247391    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beb8592b-6036-475a-b404-6acdf015fab2-istiod-ca-cert\") pod \"e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7\" (UID: \"beb8592b-6036-475a-b404-6acdf015fab2\") " pod="openshift-ingress/e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7"
Sep 14 10:03:35.247497 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:35.247428    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beb8592b-6036-475a-b404-6acdf015fab2-istio-data\") pod \"e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7\" (UID: \"beb8592b-6036-475a-b404-6acdf015fab2\") " pod="openshift-ingress/e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7"
Sep 14 10:03:35.348268 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:35.348154    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beb8592b-6036-475a-b404-6acdf015fab2-istio-token\") pod \"e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7\" (UID: \"beb8592b-6036-475a-b404-6acdf015fab2\") " pod="openshift-ingress/e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7"
Sep 14 10:03:35.348268 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:35.348213    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beb8592b-6036-475a-b404-6acdf015fab2-istio-envoy\") pod \"e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7\" (UID: \"beb8592b-6036-475a-b404-6acdf015fab2\") " pod="openshift-ingress/e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7"
Sep 14 10:03:35.348268 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:35.348247    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq4zj\" (UniqueName: \"kubernetes.io/projected/beb8592b-6036-475a-b404-6acdf015fab2-kube-api-access-bq4zj\") pod \"e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7\" (UID: \"beb8592b-6036-475a-b404-6acdf015fab2\") " pod="openshift-ingress/e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7"
Sep 14 10:03:35.348572 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:35.348276    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beb8592b-6036-475a-b404-6acdf015fab2-istiod-ca-cert\") pod \"e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7\" (UID: \"beb8592b-6036-475a-b404-6acdf015fab2\") " pod="openshift-ingress/e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7"
Sep 14 10:03:35.348572 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:35.348299    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beb8592b-6036-475a-b404-6acdf015fab2-istio-data\") pod \"e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7\" (UID: \"beb8592b-6036-475a-b404-6acdf015fab2\") " pod="openshift-ingress/e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7"
Sep 14 10:03:35.348572 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:35.348351    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beb8592b-6036-475a-b404-6acdf015fab2-workload-certs\") pod \"e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7\" (UID: \"beb8592b-6036-475a-b404-6acdf015fab2\") " pod="openshift-ingress/e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7"
Sep 14 10:03:35.348572 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:35.348381    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beb8592b-6036-475a-b404-6acdf015fab2-workload-socket\") pod \"e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7\" (UID: \"beb8592b-6036-475a-b404-6acdf015fab2\") " pod="openshift-ingress/e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7"
Sep 14 10:03:35.348572 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:35.348411    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beb8592b-6036-475a-b404-6acdf015fab2-credential-socket\") pod \"e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7\" (UID: \"beb8592b-6036-475a-b404-6acdf015fab2\") " pod="openshift-ingress/e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7"
Sep 14 10:03:35.348572 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:35.348445    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beb8592b-6036-475a-b404-6acdf015fab2-istio-podinfo\") pod \"e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7\" (UID: \"beb8592b-6036-475a-b404-6acdf015fab2\") " pod="openshift-ingress/e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7"
Sep 14 10:03:35.348897 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:35.348782    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beb8592b-6036-475a-b404-6acdf015fab2-istio-data\") pod \"e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7\" (UID: \"beb8592b-6036-475a-b404-6acdf015fab2\") " pod="openshift-ingress/e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7"
Sep 14 10:03:35.348978 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:35.348950    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beb8592b-6036-475a-b404-6acdf015fab2-workload-socket\") pod \"e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7\" (UID: \"beb8592b-6036-475a-b404-6acdf015fab2\") " pod="openshift-ingress/e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7"
Sep 14 10:03:35.349072 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:35.348993    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beb8592b-6036-475a-b404-6acdf015fab2-credential-socket\") pod \"e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7\" (UID: \"beb8592b-6036-475a-b404-6acdf015fab2\") " pod="openshift-ingress/e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7"
Sep 14 10:03:35.349072 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:35.348951    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beb8592b-6036-475a-b404-6acdf015fab2-workload-certs\") pod \"e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7\" (UID: \"beb8592b-6036-475a-b404-6acdf015fab2\") " pod="openshift-ingress/e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7"
Sep 14 10:03:35.349221 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:35.349201    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beb8592b-6036-475a-b404-6acdf015fab2-istiod-ca-cert\") pod \"e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7\" (UID: \"beb8592b-6036-475a-b404-6acdf015fab2\") " pod="openshift-ingress/e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7"
Sep 14 10:03:35.350741 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:35.350708    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beb8592b-6036-475a-b404-6acdf015fab2-istio-envoy\") pod \"e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7\" (UID: \"beb8592b-6036-475a-b404-6acdf015fab2\") " pod="openshift-ingress/e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7"
Sep 14 10:03:35.350963 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:35.350942    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beb8592b-6036-475a-b404-6acdf015fab2-istio-podinfo\") pod \"e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7\" (UID: \"beb8592b-6036-475a-b404-6acdf015fab2\") " pod="openshift-ingress/e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7"
Sep 14 10:03:35.358616 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:35.358579    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beb8592b-6036-475a-b404-6acdf015fab2-istio-token\") pod \"e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7\" (UID: \"beb8592b-6036-475a-b404-6acdf015fab2\") " pod="openshift-ingress/e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7"
Sep 14 10:03:35.358772 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:35.358691    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq4zj\" (UniqueName: \"kubernetes.io/projected/beb8592b-6036-475a-b404-6acdf015fab2-kube-api-access-bq4zj\") pod \"e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7\" (UID: \"beb8592b-6036-475a-b404-6acdf015fab2\") " pod="openshift-ingress/e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7"
Sep 14 10:03:35.364542 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:35.364505    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7"
Sep 14 10:03:35.492795 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:35.492756    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:03:35.649842 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:35.649808    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7"]
Sep 14 10:03:35.651348 ip-10-0-132-4 kubenswrapper[2563]: W0914 10:03:35.651316    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeb8592b_6036_475a_b404_6acdf015fab2.slice/crio-f0cb02e145830c5dd6dcb6d4571ecfdefe221e5c3fb16aebd7844d2f14a95c44 WatchSource:0}: Error finding container f0cb02e145830c5dd6dcb6d4571ecfdefe221e5c3fb16aebd7844d2f14a95c44: Status 404 returned error can't find the container with id f0cb02e145830c5dd6dcb6d4571ecfdefe221e5c3fb16aebd7844d2f14a95c44
Sep 14 10:03:35.660888 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:35.655635    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 10:03:35.660888 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:35.656103    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 10:03:35.660888 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:35.656487    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 10:03:35.811967 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:35.811931    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:03:36.667263 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:36.667215    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7" event={"ID":"beb8592b-6036-475a-b404-6acdf015fab2","Type":"ContainerStarted","Data":"f4374eedeb348888a77686228684c05ef1b502504e6f8b380efed384a5d39b64"}
Sep 14 10:03:36.667263 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:36.667272    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7" event={"ID":"beb8592b-6036-475a-b404-6acdf015fab2","Type":"ContainerStarted","Data":"f0cb02e145830c5dd6dcb6d4571ecfdefe221e5c3fb16aebd7844d2f14a95c44"}
Sep 14 10:03:36.699288 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:36.699218    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7" podStartSLOduration=1.699198681 podStartE2EDuration="1.699198681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:03:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:03:36.696937151 +0000 UTC m=+1205.736611772" watchObservedRunningTime="2026-09-14 10:03:36.699198681 +0000 UTC m=+1205.738873374"
Sep 14 10:03:37.365092 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:37.365048    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7"
Sep 14 10:03:37.370727 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:37.370697    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7"
Sep 14 10:03:37.674369 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:37.674272    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7"
Sep 14 10:03:37.675502 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:37.675477    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7"
Sep 14 10:03:37.890789 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:37.890754    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:03:38.633504 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:38.633472    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:03:38.728064 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:38.727992    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:03:39.077563 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:39.077528    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:03:40.100320 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:40.100287    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:03:41.220328 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:41.220292    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:03:42.058881 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:42.058842    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:03:42.988265 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:42.988231    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:03:43.799368 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:43.799330    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:03:43.874138 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:43.874102    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:03:44.043403 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:44.043358    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:03:44.355765 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:44.355715    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:03:44.459274 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:44.459237    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:03:53.177776 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:53.177729    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:03:54.821417 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:54.821377    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:03:59.406774 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:59.406702    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-0e60f3/test-model-0e60f3-kserve-84c9545bb9-bx4l2"]
Sep 14 10:03:59.414416 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:03:59.414381    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:03.314148 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:03.314092    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0e60f3-5bf29"]
Sep 14 10:04:03.317768 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:03.317734    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0e60f3-5bf29"
Sep 14 10:04:03.320338 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:03.320287    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gf4xj\""
Sep 14 10:04:03.326226 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:03.325886    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 14 10:04:03.334984 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:03.334943    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0e60f3-5bf29"]
Sep 14 10:04:03.461557 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:03.461518    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a82e3d2f-7353-442f-a760-219c672b4733-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0e60f3-5bf29\" (UID: \"a82e3d2f-7353-442f-a760-219c672b4733\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0e60f3-5bf29"
Sep 14 10:04:03.562295 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:03.562249    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a82e3d2f-7353-442f-a760-219c672b4733-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0e60f3-5bf29\" (UID: \"a82e3d2f-7353-442f-a760-219c672b4733\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0e60f3-5bf29"
Sep 14 10:04:03.562911 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:03.562890    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a82e3d2f-7353-442f-a760-219c672b4733-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0e60f3-5bf29\" (UID: \"a82e3d2f-7353-442f-a760-219c672b4733\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0e60f3-5bf29"
Sep 14 10:04:03.616235 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:03.616141    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:03.636233 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:03.636179    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0e60f3-5bf29"
Sep 14 10:04:03.949485 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:03.949455    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0e60f3-5bf29"]
Sep 14 10:04:03.952476 ip-10-0-132-4 kubenswrapper[2563]: W0914 10:04:03.952440    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda82e3d2f_7353_442f_a760_219c672b4733.slice/crio-7cc5e9733c7f1d6985f9d240e2d67b2bcd77ec64bdab21b4fdec612a8c465a2c WatchSource:0}: Error finding container 7cc5e9733c7f1d6985f9d240e2d67b2bcd77ec64bdab21b4fdec612a8c465a2c: Status 404 returned error can't find the container with id 7cc5e9733c7f1d6985f9d240e2d67b2bcd77ec64bdab21b4fdec612a8c465a2c
Sep 14 10:04:04.840613 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:04.840397    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:04.865886 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:04.865843    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0e60f3-5bf29" event={"ID":"a82e3d2f-7353-442f-a760-219c672b4733","Type":"ContainerStarted","Data":"aeaab86acb8b1d09a11f965334d95c4812a92b65e48bdfdbf63879aa30b8bf16"}
Sep 14 10:04:04.866111 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:04.865916    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0e60f3-5bf29" event={"ID":"a82e3d2f-7353-442f-a760-219c672b4733","Type":"ContainerStarted","Data":"7cc5e9733c7f1d6985f9d240e2d67b2bcd77ec64bdab21b4fdec612a8c465a2c"}
Sep 14 10:04:04.900785 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:04.900711    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0e60f3-5bf29" podStartSLOduration=1.900692996 podStartE2EDuration="1.900692996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:04:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:04:04.897569964 +0000 UTC m=+1233.937244675" watchObservedRunningTime="2026-09-14 10:04:04.900692996 +0000 UTC m=+1233.940367667"
Sep 14 10:04:05.682603 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:05.682565    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:05.875226 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:05.875186    2563 generic.go:356] "Generic (PLEG): container finished" podID="a82e3d2f-7353-442f-a760-219c672b4733" containerID="aeaab86acb8b1d09a11f965334d95c4812a92b65e48bdfdbf63879aa30b8bf16" exitCode=0
Sep 14 10:04:05.875653 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:05.875273    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0e60f3-5bf29" event={"ID":"a82e3d2f-7353-442f-a760-219c672b4733","Type":"ContainerDied","Data":"aeaab86acb8b1d09a11f965334d95c4812a92b65e48bdfdbf63879aa30b8bf16"}
Sep 14 10:04:06.649372 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:06.649274    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:07.049123 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:07.049092    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0e60f3-5bf29"
Sep 14 10:04:07.106358 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:07.106324    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a82e3d2f-7353-442f-a760-219c672b4733-maas-api-service-ca\") pod \"a82e3d2f-7353-442f-a760-219c672b4733\" (UID: \"a82e3d2f-7353-442f-a760-219c672b4733\") "
Sep 14 10:04:07.106698 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:07.106674    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a82e3d2f-7353-442f-a760-219c672b4733-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a82e3d2f-7353-442f-a760-219c672b4733" (UID: "a82e3d2f-7353-442f-a760-219c672b4733"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:04:07.208152 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:07.208045    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a82e3d2f-7353-442f-a760-219c672b4733-maas-api-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:04:07.549815 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:07.549776    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:07.892367 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:07.892275    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0e60f3-5bf29"
Sep 14 10:04:07.892367 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:07.892310    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0e60f3-5bf29" event={"ID":"a82e3d2f-7353-442f-a760-219c672b4733","Type":"ContainerDied","Data":"7cc5e9733c7f1d6985f9d240e2d67b2bcd77ec64bdab21b4fdec612a8c465a2c"}
Sep 14 10:04:07.892367 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:07.892365    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7cc5e9733c7f1d6985f9d240e2d67b2bcd77ec64bdab21b4fdec612a8c465a2c"
Sep 14 10:04:08.715231 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:08.715179    2563 patch_prober.go:28] interesting pod/e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 14 10:04:08.715761 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:08.715264    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7" podUID="beb8592b-6036-475a-b404-6acdf015fab2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 14 10:04:09.446491 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:09.446449    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:09.705548 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:09.705469    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:09.907760 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:09.907678    2563 generic.go:356] "Generic (PLEG): container finished" podID="beb8592b-6036-475a-b404-6acdf015fab2" containerID="f4374eedeb348888a77686228684c05ef1b502504e6f8b380efed384a5d39b64" exitCode=0
Sep 14 10:04:09.908170 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:09.907757    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7" event={"ID":"beb8592b-6036-475a-b404-6acdf015fab2","Type":"ContainerDied","Data":"f4374eedeb348888a77686228684c05ef1b502504e6f8b380efed384a5d39b64"}
Sep 14 10:04:09.908244 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:09.908227    2563 scope.go:117] "RemoveContainer" containerID="f4374eedeb348888a77686228684c05ef1b502504e6f8b380efed384a5d39b64"
Sep 14 10:04:09.909533 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:09.909505    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 10:04:09.909595 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:09.909582    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 10:04:09.909633 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:09.909620    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 10:04:10.519384 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:10.519345    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:10.918159 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:10.918055    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7" event={"ID":"beb8592b-6036-475a-b404-6acdf015fab2","Type":"ContainerStarted","Data":"76434b7081486bd266a3f77de96772691feba530594750732057153889a48a8d"}
Sep 14 10:04:11.348965 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:11.348901    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-24873d/test-model-24873d-kserve-dff86779c-29lrm"]
Sep 14 10:04:11.365695 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:11.365654    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7"
Sep 14 10:04:11.365900 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:11.365863    2563 patch_prober.go:28] interesting pod/e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Sep 14 10:04:11.365970 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:11.365913    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7" podUID="beb8592b-6036-475a-b404-6acdf015fab2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Sep 14 10:04:11.420511 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:11.420465    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:11.613846 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:11.613748    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:11.955752 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:11.955629    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-2901ec/e2e-conflict-b-c109e646-kserve-7d9d8884d9-b6x76"]
Sep 14 10:04:11.958170 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:11.958123    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:12.365234 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:12.365188    2563 patch_prober.go:28] interesting pod/e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Sep 14 10:04:12.365423 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:12.365248    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7" podUID="beb8592b-6036-475a-b404-6acdf015fab2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Sep 14 10:04:12.385775 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:12.385740    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:13.150113 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:13.150071    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:13.365326 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:13.365282    2563 patch_prober.go:28] interesting pod/e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Sep 14 10:04:13.365521 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:13.365347    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7" podUID="beb8592b-6036-475a-b404-6acdf015fab2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Sep 14 10:04:13.669379 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:13.669341    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:13.771972 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:13.771929    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:13.873385 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:13.873338    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:14.017740 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:14.017704    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:14.113993 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:14.113953    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:14.216735 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:14.216702    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:14.365874 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:14.365775    2563 patch_prober.go:28] interesting pod/e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Sep 14 10:04:14.365874 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:14.365843    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7" podUID="beb8592b-6036-475a-b404-6acdf015fab2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Sep 14 10:04:14.447400 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:14.447351    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-0e60f3/test-model-0e60f3-kserve-84c9545bb9-bx4l2" podUID="90443638-9dd8-4c4d-812f-146cba869af7" containerName="main" containerID="cri-o://b658770c20270cbbb20fa5b41b339d2e5764b3719099f9d47c8a8061966c23cf" gracePeriod=45
Sep 14 10:04:14.521860 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:14.521802    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7678-f90690"]
Sep 14 10:04:14.524126 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:14.522512    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a82e3d2f-7353-442f-a760-219c672b4733" containerName="revoke-keys"
Sep 14 10:04:14.524126 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:14.522538    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a82e3d2f-7353-442f-a760-219c672b4733" containerName="revoke-keys"
Sep 14 10:04:14.524126 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:14.522648    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a82e3d2f-7353-442f-a760-219c672b4733" containerName="revoke-keys"
Sep 14 10:04:14.525577 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:14.525491    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7678-f90690"
Sep 14 10:04:14.530622 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:14.529880    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-hjc9w\""
Sep 14 10:04:14.535864 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:14.535815    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7678-f90690"]
Sep 14 10:04:14.596150 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:14.596050    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-282k4\" (UniqueName: \"kubernetes.io/projected/536e8cf1-ee1d-4023-a976-fc12f5576db6-kube-api-access-282k4\") pod \"test-curl-7678-f90690\" (UID: \"536e8cf1-ee1d-4023-a976-fc12f5576db6\") " pod="opendatahub/test-curl-7678-f90690"
Sep 14 10:04:14.697088 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:14.696987    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-282k4\" (UniqueName: \"kubernetes.io/projected/536e8cf1-ee1d-4023-a976-fc12f5576db6-kube-api-access-282k4\") pod \"test-curl-7678-f90690\" (UID: \"536e8cf1-ee1d-4023-a976-fc12f5576db6\") " pod="opendatahub/test-curl-7678-f90690"
Sep 14 10:04:14.714990 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:14.714961    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-282k4\" (UniqueName: \"kubernetes.io/projected/536e8cf1-ee1d-4023-a976-fc12f5576db6-kube-api-access-282k4\") pod \"test-curl-7678-f90690\" (UID: \"536e8cf1-ee1d-4023-a976-fc12f5576db6\") " pod="opendatahub/test-curl-7678-f90690"
Sep 14 10:04:14.753829 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:14.753799    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0e60f3/test-model-0e60f3-kserve-84c9545bb9-bx4l2"
Sep 14 10:04:14.755267 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:14.755241    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:14.798302 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:14.798256    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90443638-9dd8-4c4d-812f-146cba869af7-tmp-dir\") pod \"90443638-9dd8-4c4d-812f-146cba869af7\" (UID: \"90443638-9dd8-4c4d-812f-146cba869af7\") "
Sep 14 10:04:14.798554 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:14.798377    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90443638-9dd8-4c4d-812f-146cba869af7-home\") pod \"90443638-9dd8-4c4d-812f-146cba869af7\" (UID: \"90443638-9dd8-4c4d-812f-146cba869af7\") "
Sep 14 10:04:14.798554 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:14.798437    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90443638-9dd8-4c4d-812f-146cba869af7-tls-certs\") pod \"90443638-9dd8-4c4d-812f-146cba869af7\" (UID: \"90443638-9dd8-4c4d-812f-146cba869af7\") "
Sep 14 10:04:14.798554 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:14.798467    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90443638-9dd8-4c4d-812f-146cba869af7-model-cache\") pod \"90443638-9dd8-4c4d-812f-146cba869af7\" (UID: \"90443638-9dd8-4c4d-812f-146cba869af7\") "
Sep 14 10:04:14.798799 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:14.798569    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90443638-9dd8-4c4d-812f-146cba869af7-dshm\") pod \"90443638-9dd8-4c4d-812f-146cba869af7\" (UID: \"90443638-9dd8-4c4d-812f-146cba869af7\") "
Sep 14 10:04:14.798799 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:14.798607    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7qvrj\" (UniqueName: \"kubernetes.io/projected/90443638-9dd8-4c4d-812f-146cba869af7-kube-api-access-7qvrj\") pod \"90443638-9dd8-4c4d-812f-146cba869af7\" (UID: \"90443638-9dd8-4c4d-812f-146cba869af7\") "
Sep 14 10:04:14.799587 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:14.799497    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90443638-9dd8-4c4d-812f-146cba869af7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "90443638-9dd8-4c4d-812f-146cba869af7" (UID: "90443638-9dd8-4c4d-812f-146cba869af7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:04:14.799748 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:14.799722    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90443638-9dd8-4c4d-812f-146cba869af7-model-cache" (OuterVolumeSpecName: "model-cache") pod "90443638-9dd8-4c4d-812f-146cba869af7" (UID: "90443638-9dd8-4c4d-812f-146cba869af7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:04:14.799813 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:14.799711    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90443638-9dd8-4c4d-812f-146cba869af7-home" (OuterVolumeSpecName: "home") pod "90443638-9dd8-4c4d-812f-146cba869af7" (UID: "90443638-9dd8-4c4d-812f-146cba869af7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:04:14.802592 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:14.802151    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90443638-9dd8-4c4d-812f-146cba869af7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "90443638-9dd8-4c4d-812f-146cba869af7" (UID: "90443638-9dd8-4c4d-812f-146cba869af7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:04:14.802592 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:14.802177    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90443638-9dd8-4c4d-812f-146cba869af7-kube-api-access-7qvrj" (OuterVolumeSpecName: "kube-api-access-7qvrj") pod "90443638-9dd8-4c4d-812f-146cba869af7" (UID: "90443638-9dd8-4c4d-812f-146cba869af7"). InnerVolumeSpecName "kube-api-access-7qvrj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:04:14.802592 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:14.802536    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90443638-9dd8-4c4d-812f-146cba869af7-dshm" (OuterVolumeSpecName: "dshm") pod "90443638-9dd8-4c4d-812f-146cba869af7" (UID: "90443638-9dd8-4c4d-812f-146cba869af7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:04:14.848086 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:14.847693    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7678-f90690"
Sep 14 10:04:14.900206 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:14.900110    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qvrj\" (UniqueName: \"kubernetes.io/projected/90443638-9dd8-4c4d-812f-146cba869af7-kube-api-access-7qvrj\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:04:14.900206 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:14.900147    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90443638-9dd8-4c4d-812f-146cba869af7-tmp-dir\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:04:14.900206 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:14.900157    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90443638-9dd8-4c4d-812f-146cba869af7-home\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:04:14.900206 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:14.900165    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90443638-9dd8-4c4d-812f-146cba869af7-tls-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:04:14.900206 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:14.900174    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90443638-9dd8-4c4d-812f-146cba869af7-model-cache\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:04:14.900206 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:14.900182    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90443638-9dd8-4c4d-812f-146cba869af7-dshm\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:04:14.962055 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:14.962018    2563 generic.go:356] "Generic (PLEG): container finished" podID="90443638-9dd8-4c4d-812f-146cba869af7" containerID="b658770c20270cbbb20fa5b41b339d2e5764b3719099f9d47c8a8061966c23cf" exitCode=0
Sep 14 10:04:14.962236 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:14.962067    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0e60f3/test-model-0e60f3-kserve-84c9545bb9-bx4l2"
Sep 14 10:04:14.962236 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:14.962145    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0e60f3/test-model-0e60f3-kserve-84c9545bb9-bx4l2" event={"ID":"90443638-9dd8-4c4d-812f-146cba869af7","Type":"ContainerDied","Data":"b658770c20270cbbb20fa5b41b339d2e5764b3719099f9d47c8a8061966c23cf"}
Sep 14 10:04:14.962236 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:14.962174    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0e60f3/test-model-0e60f3-kserve-84c9545bb9-bx4l2" event={"ID":"90443638-9dd8-4c4d-812f-146cba869af7","Type":"ContainerDied","Data":"77f775bb3d18b233a57588c976ad37803f250e27225acddf0a4c929a0c1c66ae"}
Sep 14 10:04:14.962236 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:14.962190    2563 scope.go:117] "RemoveContainer" containerID="b658770c20270cbbb20fa5b41b339d2e5764b3719099f9d47c8a8061966c23cf"
Sep 14 10:04:14.974171 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:14.974140    2563 scope.go:117] "RemoveContainer" containerID="b658770c20270cbbb20fa5b41b339d2e5764b3719099f9d47c8a8061966c23cf"
Sep 14 10:04:14.974557 ip-10-0-132-4 kubenswrapper[2563]: E0914 10:04:14.974535    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b658770c20270cbbb20fa5b41b339d2e5764b3719099f9d47c8a8061966c23cf\": container with ID starting with b658770c20270cbbb20fa5b41b339d2e5764b3719099f9d47c8a8061966c23cf not found: ID does not exist" containerID="b658770c20270cbbb20fa5b41b339d2e5764b3719099f9d47c8a8061966c23cf"
Sep 14 10:04:14.974640 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:14.974568    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b658770c20270cbbb20fa5b41b339d2e5764b3719099f9d47c8a8061966c23cf"} err="failed to get container status \"b658770c20270cbbb20fa5b41b339d2e5764b3719099f9d47c8a8061966c23cf\": rpc error: code = NotFound desc = could not find container \"b658770c20270cbbb20fa5b41b339d2e5764b3719099f9d47c8a8061966c23cf\": container with ID starting with b658770c20270cbbb20fa5b41b339d2e5764b3719099f9d47c8a8061966c23cf not found: ID does not exist"
Sep 14 10:04:14.994012 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:14.993951    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-0e60f3/test-model-0e60f3-kserve-84c9545bb9-bx4l2"]
Sep 14 10:04:15.003849 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:15.003810    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-0e60f3/test-model-0e60f3-kserve-84c9545bb9-bx4l2"]
Sep 14 10:04:15.131959 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:15.130094    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7678-f90690"]
Sep 14 10:04:15.134770 ip-10-0-132-4 kubenswrapper[2563]: W0914 10:04:15.134723    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod536e8cf1_ee1d_4023_a976_fc12f5576db6.slice/crio-b1cab048e49f0f51f2e5b02287ebf3901e182f8e05905fdb997693d07d1500b7 WatchSource:0}: Error finding container b1cab048e49f0f51f2e5b02287ebf3901e182f8e05905fdb997693d07d1500b7: Status 404 returned error can't find the container with id b1cab048e49f0f51f2e5b02287ebf3901e182f8e05905fdb997693d07d1500b7
Sep 14 10:04:15.192329 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:15.192297    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:15.365191 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:15.365148    2563 patch_prober.go:28] interesting pod/e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Sep 14 10:04:15.365610 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:15.365219    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7" podUID="beb8592b-6036-475a-b404-6acdf015fab2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Sep 14 10:04:15.483477 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:15.483372    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90443638-9dd8-4c4d-812f-146cba869af7" path="/var/lib/kubelet/pods/90443638-9dd8-4c4d-812f-146cba869af7/volumes"
Sep 14 10:04:15.970502 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:15.970458    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7678-f90690" event={"ID":"536e8cf1-ee1d-4023-a976-fc12f5576db6","Type":"ContainerStarted","Data":"dee2e2acab16bd50fc8e0621fdd3a51158770a470a5b51ce4707af6ec040fb9d"}
Sep 14 10:04:15.970502 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:15.970518    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7678-f90690" event={"ID":"536e8cf1-ee1d-4023-a976-fc12f5576db6","Type":"ContainerStarted","Data":"b1cab048e49f0f51f2e5b02287ebf3901e182f8e05905fdb997693d07d1500b7"}
Sep 14 10:04:15.987067 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:15.986970    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7678-f90690" podStartSLOduration=1.986949337 podStartE2EDuration="1.986949337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:04:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:04:15.986226512 +0000 UTC m=+1245.025901132" watchObservedRunningTime="2026-09-14 10:04:15.986949337 +0000 UTC m=+1245.026623960"
Sep 14 10:04:16.226528 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:16.226430    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:16.430330 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:16.430298    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7"
Sep 14 10:04:16.430836 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:16.430656    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7"
Sep 14 10:04:16.432089 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:16.432062    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7"
Sep 14 10:04:16.639321 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:16.639271    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7678-f90690"]
Sep 14 10:04:16.644743 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:16.644706    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7678-f90690"]
Sep 14 10:04:16.758788 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:16.758741    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7678-1dfe54"]
Sep 14 10:04:16.759175 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:16.759161    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90443638-9dd8-4c4d-812f-146cba869af7" containerName="main"
Sep 14 10:04:16.759240 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:16.759178    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="90443638-9dd8-4c4d-812f-146cba869af7" containerName="main"
Sep 14 10:04:16.759240 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:16.759197    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="536e8cf1-ee1d-4023-a976-fc12f5576db6" containerName="test-curl-7678-f90690"
Sep 14 10:04:16.759240 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:16.759203    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="536e8cf1-ee1d-4023-a976-fc12f5576db6" containerName="test-curl-7678-f90690"
Sep 14 10:04:16.759338 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:16.759278    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="536e8cf1-ee1d-4023-a976-fc12f5576db6" containerName="test-curl-7678-f90690"
Sep 14 10:04:16.759338 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:16.759290    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="90443638-9dd8-4c4d-812f-146cba869af7" containerName="main"
Sep 14 10:04:16.761676 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:16.761655    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7678-1dfe54"
Sep 14 10:04:16.769415 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:16.769364    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7678-1dfe54"]
Sep 14 10:04:16.819241 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:16.819197    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h946\" (UniqueName: \"kubernetes.io/projected/c0e306ab-d0ba-4a06-9959-e169e93bc2e2-kube-api-access-6h946\") pod \"test-curl-7678-1dfe54\" (UID: \"c0e306ab-d0ba-4a06-9959-e169e93bc2e2\") " pod="opendatahub/test-curl-7678-1dfe54"
Sep 14 10:04:16.920540 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:16.920445    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h946\" (UniqueName: \"kubernetes.io/projected/c0e306ab-d0ba-4a06-9959-e169e93bc2e2-kube-api-access-6h946\") pod \"test-curl-7678-1dfe54\" (UID: \"c0e306ab-d0ba-4a06-9959-e169e93bc2e2\") " pod="opendatahub/test-curl-7678-1dfe54"
Sep 14 10:04:16.929502 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:16.929470    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h946\" (UniqueName: \"kubernetes.io/projected/c0e306ab-d0ba-4a06-9959-e169e93bc2e2-kube-api-access-6h946\") pod \"test-curl-7678-1dfe54\" (UID: \"c0e306ab-d0ba-4a06-9959-e169e93bc2e2\") " pod="opendatahub/test-curl-7678-1dfe54"
Sep 14 10:04:16.981552 ip-10-0-132-4 kubenswrapper[2563]: E0914 10:04:16.981514    2563 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7678-f90690" container="test-curl-7678-f90690"
Sep 14 10:04:16.995260 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:16.995202    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:16.997104 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:16.997066    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0e60f3-29823000-9r5gg"]
Sep 14 10:04:16.999806 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:16.999777    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0e60f3-29823000-9r5gg"]
Sep 14 10:04:17.075687 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:17.075653    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7678-1dfe54"
Sep 14 10:04:17.086232 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:17.086197    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:17.186155 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:17.186059    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:17.241359 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:17.241319    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0e60f3-5bf29"]
Sep 14 10:04:17.244303 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:17.244271    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0e60f3-5bf29"]
Sep 14 10:04:17.257742 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:17.257705    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:17.359294 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:17.359262    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7678-1dfe54"]
Sep 14 10:04:17.361648 ip-10-0-132-4 kubenswrapper[2563]: W0914 10:04:17.361618    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc0e306ab_d0ba_4a06_9959_e169e93bc2e2.slice/crio-dc5a93c8bf40668fdca7f8aed3e38dd27f66130d16c6c291a49da0f6d9f00473 WatchSource:0}: Error finding container dc5a93c8bf40668fdca7f8aed3e38dd27f66130d16c6c291a49da0f6d9f00473: Status 404 returned error can't find the container with id dc5a93c8bf40668fdca7f8aed3e38dd27f66130d16c6c291a49da0f6d9f00473
Sep 14 10:04:17.482908 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:17.482803    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45f8f6ed-1ae8-4268-9224-6c58faa23fee" path="/var/lib/kubelet/pods/45f8f6ed-1ae8-4268-9224-6c58faa23fee/volumes"
Sep 14 10:04:17.483392 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:17.483356    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a82e3d2f-7353-442f-a760-219c672b4733" path="/var/lib/kubelet/pods/a82e3d2f-7353-442f-a760-219c672b4733/volumes"
Sep 14 10:04:17.650675 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:17.650637    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:17.752204 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:17.752170    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:17.990157 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:17.990115    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7678-1dfe54" event={"ID":"c0e306ab-d0ba-4a06-9959-e169e93bc2e2","Type":"ContainerStarted","Data":"5824f759f0e842c6a4d2a5bb8ecafcc59bdbf1f49386ee206fc8c0e9e54e1822"}
Sep 14 10:04:17.990375 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:17.990167    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7678-1dfe54" event={"ID":"c0e306ab-d0ba-4a06-9959-e169e93bc2e2","Type":"ContainerStarted","Data":"dc5a93c8bf40668fdca7f8aed3e38dd27f66130d16c6c291a49da0f6d9f00473"}
Sep 14 10:04:17.990476 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:17.990453    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7678-f90690" podUID="536e8cf1-ee1d-4023-a976-fc12f5576db6" containerName="test-curl-7678-f90690" containerID="cri-o://dee2e2acab16bd50fc8e0621fdd3a51158770a470a5b51ce4707af6ec040fb9d" gracePeriod=2
Sep 14 10:04:18.006492 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:18.006380    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7678-1dfe54" podStartSLOduration=2.00635978 podStartE2EDuration="2.00635978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:04:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:04:18.003871476 +0000 UTC m=+1247.043546097" watchObservedRunningTime="2026-09-14 10:04:18.00635978 +0000 UTC m=+1247.046034400"
Sep 14 10:04:18.539114 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:18.539075    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:18.749265 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:18.749165    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7678-1dfe54"]
Sep 14 10:04:18.753780 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:18.753740    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7678-1dfe54"]
Sep 14 10:04:18.889947 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:18.889882    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7678-03405c"]
Sep 14 10:04:18.890391 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:18.890374    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0e306ab-d0ba-4a06-9959-e169e93bc2e2" containerName="test-curl-7678-1dfe54"
Sep 14 10:04:18.890442 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:18.890394    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0e306ab-d0ba-4a06-9959-e169e93bc2e2" containerName="test-curl-7678-1dfe54"
Sep 14 10:04:18.890479 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:18.890464    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0e306ab-d0ba-4a06-9959-e169e93bc2e2" containerName="test-curl-7678-1dfe54"
Sep 14 10:04:18.893517 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:18.893491    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7678-03405c"
Sep 14 10:04:18.900468 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:18.900437    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7678-03405c"]
Sep 14 10:04:18.942365 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:18.942323    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frjk6\" (UniqueName: \"kubernetes.io/projected/8505d095-7937-4ae1-a3ca-0e1fb59b3049-kube-api-access-frjk6\") pod \"test-curl-7678-03405c\" (UID: \"8505d095-7937-4ae1-a3ca-0e1fb59b3049\") " pod="opendatahub/test-curl-7678-03405c"
Sep 14 10:04:18.997385 ip-10-0-132-4 kubenswrapper[2563]: E0914 10:04:18.997339    2563 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7678-1dfe54" container="test-curl-7678-1dfe54"
Sep 14 10:04:19.043432 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:19.043321    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frjk6\" (UniqueName: \"kubernetes.io/projected/8505d095-7937-4ae1-a3ca-0e1fb59b3049-kube-api-access-frjk6\") pod \"test-curl-7678-03405c\" (UID: \"8505d095-7937-4ae1-a3ca-0e1fb59b3049\") " pod="opendatahub/test-curl-7678-03405c"
Sep 14 10:04:19.051488 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:19.051452    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frjk6\" (UniqueName: \"kubernetes.io/projected/8505d095-7937-4ae1-a3ca-0e1fb59b3049-kube-api-access-frjk6\") pod \"test-curl-7678-03405c\" (UID: \"8505d095-7937-4ae1-a3ca-0e1fb59b3049\") " pod="opendatahub/test-curl-7678-03405c"
Sep 14 10:04:19.160261 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:19.160220    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:19.205349 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:19.205308    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7678-03405c"
Sep 14 10:04:19.383671 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:19.383580    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:19.488040 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:19.487987    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7678-03405c"]
Sep 14 10:04:19.490775 ip-10-0-132-4 kubenswrapper[2563]: W0914 10:04:19.490739    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8505d095_7937_4ae1_a3ca_0e1fb59b3049.slice/crio-7d17654bda6189292e64251c900b05333500d322567aa7ed5e481982f6be5390 WatchSource:0}: Error finding container 7d17654bda6189292e64251c900b05333500d322567aa7ed5e481982f6be5390: Status 404 returned error can't find the container with id 7d17654bda6189292e64251c900b05333500d322567aa7ed5e481982f6be5390
Sep 14 10:04:20.005692 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:20.005594    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7678-03405c" event={"ID":"8505d095-7937-4ae1-a3ca-0e1fb59b3049","Type":"ContainerStarted","Data":"071a414e7d75be885a4a06604dcc5ee1137cba04504877552a66261483b949de"}
Sep 14 10:04:20.005692 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:20.005638    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7678-03405c" event={"ID":"8505d095-7937-4ae1-a3ca-0e1fb59b3049","Type":"ContainerStarted","Data":"7d17654bda6189292e64251c900b05333500d322567aa7ed5e481982f6be5390"}
Sep 14 10:04:20.006131 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:20.005778    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7678-1dfe54" podUID="c0e306ab-d0ba-4a06-9959-e169e93bc2e2" containerName="test-curl-7678-1dfe54" containerID="cri-o://5824f759f0e842c6a4d2a5bb8ecafcc59bdbf1f49386ee206fc8c0e9e54e1822" gracePeriod=2
Sep 14 10:04:20.024422 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:20.024363    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7678-03405c" podStartSLOduration=2.024344375 podStartE2EDuration="2.024344375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:04:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:04:20.020993091 +0000 UTC m=+1249.060667709" watchObservedRunningTime="2026-09-14 10:04:20.024344375 +0000 UTC m=+1249.064019001"
Sep 14 10:04:20.143592 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:20.143562    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7678-f90690"
Sep 14 10:04:20.255617 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:20.255526    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-282k4\" (UniqueName: \"kubernetes.io/projected/536e8cf1-ee1d-4023-a976-fc12f5576db6-kube-api-access-282k4\") pod \"536e8cf1-ee1d-4023-a976-fc12f5576db6\" (UID: \"536e8cf1-ee1d-4023-a976-fc12f5576db6\") "
Sep 14 10:04:20.257721 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:20.257676    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/536e8cf1-ee1d-4023-a976-fc12f5576db6-kube-api-access-282k4" (OuterVolumeSpecName: "kube-api-access-282k4") pod "536e8cf1-ee1d-4023-a976-fc12f5576db6" (UID: "536e8cf1-ee1d-4023-a976-fc12f5576db6"). InnerVolumeSpecName "kube-api-access-282k4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:04:20.357301 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:20.357257    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-282k4\" (UniqueName: \"kubernetes.io/projected/536e8cf1-ee1d-4023-a976-fc12f5576db6-kube-api-access-282k4\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:04:20.410463 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:20.410430    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:20.664794 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:20.664680    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7678-03405c"]
Sep 14 10:04:20.668442 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:20.668399    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7678-03405c"]
Sep 14 10:04:20.795164 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:20.795114    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7678-fc9942"]
Sep 14 10:04:20.795594 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:20.795576    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8505d095-7937-4ae1-a3ca-0e1fb59b3049" containerName="test-curl-7678-03405c"
Sep 14 10:04:20.795647 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:20.795595    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8505d095-7937-4ae1-a3ca-0e1fb59b3049" containerName="test-curl-7678-03405c"
Sep 14 10:04:20.795690 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:20.795679    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8505d095-7937-4ae1-a3ca-0e1fb59b3049" containerName="test-curl-7678-03405c"
Sep 14 10:04:20.798264 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:20.798242    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7678-fc9942"
Sep 14 10:04:20.807649 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:20.807619    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7678-fc9942"]
Sep 14 10:04:20.862013 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:20.861951    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65tnr\" (UniqueName: \"kubernetes.io/projected/6bef203a-7ed2-491c-ba04-f6d58da7f06b-kube-api-access-65tnr\") pod \"test-curl-7678-fc9942\" (UID: \"6bef203a-7ed2-491c-ba04-f6d58da7f06b\") " pod="opendatahub/test-curl-7678-fc9942"
Sep 14 10:04:20.963160 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:20.963054    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65tnr\" (UniqueName: \"kubernetes.io/projected/6bef203a-7ed2-491c-ba04-f6d58da7f06b-kube-api-access-65tnr\") pod \"test-curl-7678-fc9942\" (UID: \"6bef203a-7ed2-491c-ba04-f6d58da7f06b\") " pod="opendatahub/test-curl-7678-fc9942"
Sep 14 10:04:20.971656 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:20.971610    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65tnr\" (UniqueName: \"kubernetes.io/projected/6bef203a-7ed2-491c-ba04-f6d58da7f06b-kube-api-access-65tnr\") pod \"test-curl-7678-fc9942\" (UID: \"6bef203a-7ed2-491c-ba04-f6d58da7f06b\") " pod="opendatahub/test-curl-7678-fc9942"
Sep 14 10:04:21.014572 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:21.014535    2563 generic.go:356] "Generic (PLEG): container finished" podID="536e8cf1-ee1d-4023-a976-fc12f5576db6" containerID="dee2e2acab16bd50fc8e0621fdd3a51158770a470a5b51ce4707af6ec040fb9d" exitCode=137
Sep 14 10:04:21.015081 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:21.014583    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7678-f90690"
Sep 14 10:04:21.015081 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:21.014637    2563 scope.go:117] "RemoveContainer" containerID="dee2e2acab16bd50fc8e0621fdd3a51158770a470a5b51ce4707af6ec040fb9d"
Sep 14 10:04:21.015081 ip-10-0-132-4 kubenswrapper[2563]: E0914 10:04:21.014948    2563 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7678-03405c" container="test-curl-7678-03405c"
Sep 14 10:04:21.025256 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:21.025235    2563 scope.go:117] "RemoveContainer" containerID="dee2e2acab16bd50fc8e0621fdd3a51158770a470a5b51ce4707af6ec040fb9d"
Sep 14 10:04:21.025586 ip-10-0-132-4 kubenswrapper[2563]: E0914 10:04:21.025564    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dee2e2acab16bd50fc8e0621fdd3a51158770a470a5b51ce4707af6ec040fb9d\": container with ID starting with dee2e2acab16bd50fc8e0621fdd3a51158770a470a5b51ce4707af6ec040fb9d not found: ID does not exist" containerID="dee2e2acab16bd50fc8e0621fdd3a51158770a470a5b51ce4707af6ec040fb9d"
Sep 14 10:04:21.025639 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:21.025598    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dee2e2acab16bd50fc8e0621fdd3a51158770a470a5b51ce4707af6ec040fb9d"} err="failed to get container status \"dee2e2acab16bd50fc8e0621fdd3a51158770a470a5b51ce4707af6ec040fb9d\": rpc error: code = NotFound desc = could not find container \"dee2e2acab16bd50fc8e0621fdd3a51158770a470a5b51ce4707af6ec040fb9d\": container with ID starting with dee2e2acab16bd50fc8e0621fdd3a51158770a470a5b51ce4707af6ec040fb9d not found: ID does not exist"
Sep 14 10:04:21.025953 ip-10-0-132-4 kubenswrapper[2563]: E0914 10:04:21.025934    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7678-f90690\" is forbidden: User \"system:node:ip-10-0-132-4.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-4.ec2.internal' and this object" podUID="536e8cf1-ee1d-4023-a976-fc12f5576db6" pod="opendatahub/test-curl-7678-f90690"
Sep 14 10:04:21.110926 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:21.110879    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7678-fc9942"
Sep 14 10:04:21.201036 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:21.200979    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:21.391364 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:21.391332    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7678-fc9942"]
Sep 14 10:04:21.392993 ip-10-0-132-4 kubenswrapper[2563]: W0914 10:04:21.392958    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6bef203a_7ed2_491c_ba04_f6d58da7f06b.slice/crio-55193a5bc91975176ce97baf30a6dd4f0238634a1574d1f36db5167a9c4f0912 WatchSource:0}: Error finding container 55193a5bc91975176ce97baf30a6dd4f0238634a1574d1f36db5167a9c4f0912: Status 404 returned error can't find the container with id 55193a5bc91975176ce97baf30a6dd4f0238634a1574d1f36db5167a9c4f0912
Sep 14 10:04:21.481625 ip-10-0-132-4 kubenswrapper[2563]: E0914 10:04:21.481586    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7678-f90690\" is forbidden: User \"system:node:ip-10-0-132-4.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-4.ec2.internal' and this object" podUID="536e8cf1-ee1d-4023-a976-fc12f5576db6" pod="opendatahub/test-curl-7678-f90690"
Sep 14 10:04:21.484359 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:21.484329    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="536e8cf1-ee1d-4023-a976-fc12f5576db6" path="/var/lib/kubelet/pods/536e8cf1-ee1d-4023-a976-fc12f5576db6/volumes"
Sep 14 10:04:22.023496 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:22.023456    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7678-fc9942" event={"ID":"6bef203a-7ed2-491c-ba04-f6d58da7f06b","Type":"ContainerStarted","Data":"678274211bc9da16449f8cbfcdd0a75ffed894b7c7b89f546bfb78fc86d2d5a4"}
Sep 14 10:04:22.023930 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:22.023503    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7678-fc9942" event={"ID":"6bef203a-7ed2-491c-ba04-f6d58da7f06b","Type":"ContainerStarted","Data":"55193a5bc91975176ce97baf30a6dd4f0238634a1574d1f36db5167a9c4f0912"}
Sep 14 10:04:22.024753 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:22.024724    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7678-03405c" podUID="8505d095-7937-4ae1-a3ca-0e1fb59b3049" containerName="test-curl-7678-03405c" containerID="cri-o://071a414e7d75be885a4a06604dcc5ee1137cba04504877552a66261483b949de" gracePeriod=2
Sep 14 10:04:22.040484 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:22.040427    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7678-fc9942" podStartSLOduration=2.040410018 podStartE2EDuration="2.040410018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:04:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:04:22.039023365 +0000 UTC m=+1251.078697979" watchObservedRunningTime="2026-09-14 10:04:22.040410018 +0000 UTC m=+1251.080084639"
Sep 14 10:04:22.092659 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:22.091782    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:22.205853 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:22.205823    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7678-1dfe54"
Sep 14 10:04:22.278194 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:22.278084    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6h946\" (UniqueName: \"kubernetes.io/projected/c0e306ab-d0ba-4a06-9959-e169e93bc2e2-kube-api-access-6h946\") pod \"c0e306ab-d0ba-4a06-9959-e169e93bc2e2\" (UID: \"c0e306ab-d0ba-4a06-9959-e169e93bc2e2\") "
Sep 14 10:04:22.280278 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:22.280249    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0e306ab-d0ba-4a06-9959-e169e93bc2e2-kube-api-access-6h946" (OuterVolumeSpecName: "kube-api-access-6h946") pod "c0e306ab-d0ba-4a06-9959-e169e93bc2e2" (UID: "c0e306ab-d0ba-4a06-9959-e169e93bc2e2"). InnerVolumeSpecName "kube-api-access-6h946". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:04:22.379675 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:22.379629    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h946\" (UniqueName: \"kubernetes.io/projected/c0e306ab-d0ba-4a06-9959-e169e93bc2e2-kube-api-access-6h946\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:04:22.677186 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:22.677078    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7678-fc9942"]
Sep 14 10:04:22.690573 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:22.690539    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7678-fc9942"]
Sep 14 10:04:23.033278 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:23.033242    2563 generic.go:356] "Generic (PLEG): container finished" podID="c0e306ab-d0ba-4a06-9959-e169e93bc2e2" containerID="5824f759f0e842c6a4d2a5bb8ecafcc59bdbf1f49386ee206fc8c0e9e54e1822" exitCode=137
Sep 14 10:04:23.033665 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:23.033317    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7678-1dfe54"
Sep 14 10:04:23.033665 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:23.033352    2563 scope.go:117] "RemoveContainer" containerID="5824f759f0e842c6a4d2a5bb8ecafcc59bdbf1f49386ee206fc8c0e9e54e1822"
Sep 14 10:04:23.033665 ip-10-0-132-4 kubenswrapper[2563]: E0914 10:04:23.033495    2563 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7678-fc9942" container="test-curl-7678-fc9942"
Sep 14 10:04:23.044353 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:23.044326    2563 scope.go:117] "RemoveContainer" containerID="5824f759f0e842c6a4d2a5bb8ecafcc59bdbf1f49386ee206fc8c0e9e54e1822"
Sep 14 10:04:23.044487 ip-10-0-132-4 kubenswrapper[2563]: E0914 10:04:23.044323    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7678-1dfe54\" is forbidden: User \"system:node:ip-10-0-132-4.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-4.ec2.internal' and this object" podUID="c0e306ab-d0ba-4a06-9959-e169e93bc2e2" pod="opendatahub/test-curl-7678-1dfe54"
Sep 14 10:04:23.044727 ip-10-0-132-4 kubenswrapper[2563]: E0914 10:04:23.044706    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5824f759f0e842c6a4d2a5bb8ecafcc59bdbf1f49386ee206fc8c0e9e54e1822\": container with ID starting with 5824f759f0e842c6a4d2a5bb8ecafcc59bdbf1f49386ee206fc8c0e9e54e1822 not found: ID does not exist" containerID="5824f759f0e842c6a4d2a5bb8ecafcc59bdbf1f49386ee206fc8c0e9e54e1822"
Sep 14 10:04:23.044785 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:23.044741    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5824f759f0e842c6a4d2a5bb8ecafcc59bdbf1f49386ee206fc8c0e9e54e1822"} err="failed to get container status \"5824f759f0e842c6a4d2a5bb8ecafcc59bdbf1f49386ee206fc8c0e9e54e1822\": rpc error: code = NotFound desc = could not find container \"5824f759f0e842c6a4d2a5bb8ecafcc59bdbf1f49386ee206fc8c0e9e54e1822\": container with ID starting with 5824f759f0e842c6a4d2a5bb8ecafcc59bdbf1f49386ee206fc8c0e9e54e1822 not found: ID does not exist"
Sep 14 10:04:23.266432 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:23.266390    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:23.483188 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:23.483063    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0e306ab-d0ba-4a06-9959-e169e93bc2e2" path="/var/lib/kubelet/pods/c0e306ab-d0ba-4a06-9959-e169e93bc2e2/volumes"
Sep 14 10:04:23.554718 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:23.554680    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5"]
Sep 14 10:04:23.555198 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:23.555179    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bef203a-7ed2-491c-ba04-f6d58da7f06b" containerName="test-curl-7678-fc9942"
Sep 14 10:04:23.555312 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:23.555200    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bef203a-7ed2-491c-ba04-f6d58da7f06b" containerName="test-curl-7678-fc9942"
Sep 14 10:04:23.555370 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:23.555321    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bef203a-7ed2-491c-ba04-f6d58da7f06b" containerName="test-curl-7678-fc9942"
Sep 14 10:04:23.558429 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:23.558401    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5"
Sep 14 10:04:23.560560 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:23.560531    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-75017ebc-openshift-default-dockercfg-l7svt\""
Sep 14 10:04:23.571424 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:23.571388    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5"]
Sep 14 10:04:23.583531 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:23.583497    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:23.693888 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:23.693838    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg87f\" (UniqueName: \"kubernetes.io/projected/f7b2e299-2b37-4d21-9264-1cbf9d15c274-kube-api-access-cg87f\") pod \"e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5\" (UID: \"f7b2e299-2b37-4d21-9264-1cbf9d15c274\") " pod="openshift-ingress/e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5"
Sep 14 10:04:23.694135 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:23.694100    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7b2e299-2b37-4d21-9264-1cbf9d15c274-workload-certs\") pod \"e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5\" (UID: \"f7b2e299-2b37-4d21-9264-1cbf9d15c274\") " pod="openshift-ingress/e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5"
Sep 14 10:04:23.694212 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:23.694139    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7b2e299-2b37-4d21-9264-1cbf9d15c274-istio-data\") pod \"e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5\" (UID: \"f7b2e299-2b37-4d21-9264-1cbf9d15c274\") " pod="openshift-ingress/e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5"
Sep 14 10:04:23.694212 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:23.694157    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7b2e299-2b37-4d21-9264-1cbf9d15c274-istio-podinfo\") pod \"e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5\" (UID: \"f7b2e299-2b37-4d21-9264-1cbf9d15c274\") " pod="openshift-ingress/e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5"
Sep 14 10:04:23.694305 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:23.694246    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7b2e299-2b37-4d21-9264-1cbf9d15c274-workload-socket\") pod \"e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5\" (UID: \"f7b2e299-2b37-4d21-9264-1cbf9d15c274\") " pod="openshift-ingress/e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5"
Sep 14 10:04:23.694305 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:23.694276    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7b2e299-2b37-4d21-9264-1cbf9d15c274-istio-envoy\") pod \"e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5\" (UID: \"f7b2e299-2b37-4d21-9264-1cbf9d15c274\") " pod="openshift-ingress/e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5"
Sep 14 10:04:23.694403 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:23.694366    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7b2e299-2b37-4d21-9264-1cbf9d15c274-istiod-ca-cert\") pod \"e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5\" (UID: \"f7b2e299-2b37-4d21-9264-1cbf9d15c274\") " pod="openshift-ingress/e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5"
Sep 14 10:04:23.694439 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:23.694408    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7b2e299-2b37-4d21-9264-1cbf9d15c274-credential-socket\") pod \"e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5\" (UID: \"f7b2e299-2b37-4d21-9264-1cbf9d15c274\") " pod="openshift-ingress/e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5"
Sep 14 10:04:23.694497 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:23.694479    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7b2e299-2b37-4d21-9264-1cbf9d15c274-istio-token\") pod \"e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5\" (UID: \"f7b2e299-2b37-4d21-9264-1cbf9d15c274\") " pod="openshift-ingress/e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5"
Sep 14 10:04:23.795949 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:23.795905    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7b2e299-2b37-4d21-9264-1cbf9d15c274-workload-certs\") pod \"e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5\" (UID: \"f7b2e299-2b37-4d21-9264-1cbf9d15c274\") " pod="openshift-ingress/e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5"
Sep 14 10:04:23.795949 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:23.795950    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7b2e299-2b37-4d21-9264-1cbf9d15c274-istio-data\") pod \"e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5\" (UID: \"f7b2e299-2b37-4d21-9264-1cbf9d15c274\") " pod="openshift-ingress/e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5"
Sep 14 10:04:23.795949 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:23.795972    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7b2e299-2b37-4d21-9264-1cbf9d15c274-istio-podinfo\") pod \"e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5\" (UID: \"f7b2e299-2b37-4d21-9264-1cbf9d15c274\") " pod="openshift-ingress/e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5"
Sep 14 10:04:23.796334 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:23.796039    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7b2e299-2b37-4d21-9264-1cbf9d15c274-workload-socket\") pod \"e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5\" (UID: \"f7b2e299-2b37-4d21-9264-1cbf9d15c274\") " pod="openshift-ingress/e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5"
Sep 14 10:04:23.796334 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:23.796060    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7b2e299-2b37-4d21-9264-1cbf9d15c274-istio-envoy\") pod \"e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5\" (UID: \"f7b2e299-2b37-4d21-9264-1cbf9d15c274\") " pod="openshift-ingress/e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5"
Sep 14 10:04:23.796334 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:23.796104    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7b2e299-2b37-4d21-9264-1cbf9d15c274-istiod-ca-cert\") pod \"e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5\" (UID: \"f7b2e299-2b37-4d21-9264-1cbf9d15c274\") " pod="openshift-ingress/e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5"
Sep 14 10:04:23.796334 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:23.796161    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7b2e299-2b37-4d21-9264-1cbf9d15c274-credential-socket\") pod \"e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5\" (UID: \"f7b2e299-2b37-4d21-9264-1cbf9d15c274\") " pod="openshift-ingress/e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5"
Sep 14 10:04:23.796334 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:23.796286    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7b2e299-2b37-4d21-9264-1cbf9d15c274-istio-token\") pod \"e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5\" (UID: \"f7b2e299-2b37-4d21-9264-1cbf9d15c274\") " pod="openshift-ingress/e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5"
Sep 14 10:04:23.796600 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:23.796369    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg87f\" (UniqueName: \"kubernetes.io/projected/f7b2e299-2b37-4d21-9264-1cbf9d15c274-kube-api-access-cg87f\") pod \"e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5\" (UID: \"f7b2e299-2b37-4d21-9264-1cbf9d15c274\") " pod="openshift-ingress/e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5"
Sep 14 10:04:23.796600 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:23.796432    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7b2e299-2b37-4d21-9264-1cbf9d15c274-workload-certs\") pod \"e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5\" (UID: \"f7b2e299-2b37-4d21-9264-1cbf9d15c274\") " pod="openshift-ingress/e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5"
Sep 14 10:04:23.796722 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:23.796694    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7b2e299-2b37-4d21-9264-1cbf9d15c274-credential-socket\") pod \"e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5\" (UID: \"f7b2e299-2b37-4d21-9264-1cbf9d15c274\") " pod="openshift-ingress/e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5"
Sep 14 10:04:23.796790 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:23.796747    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7b2e299-2b37-4d21-9264-1cbf9d15c274-workload-socket\") pod \"e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5\" (UID: \"f7b2e299-2b37-4d21-9264-1cbf9d15c274\") " pod="openshift-ingress/e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5"
Sep 14 10:04:23.796898 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:23.796880    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7b2e299-2b37-4d21-9264-1cbf9d15c274-istio-data\") pod \"e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5\" (UID: \"f7b2e299-2b37-4d21-9264-1cbf9d15c274\") " pod="openshift-ingress/e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5"
Sep 14 10:04:23.796954 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:23.796908    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7b2e299-2b37-4d21-9264-1cbf9d15c274-istiod-ca-cert\") pod \"e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5\" (UID: \"f7b2e299-2b37-4d21-9264-1cbf9d15c274\") " pod="openshift-ingress/e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5"
Sep 14 10:04:23.798840 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:23.798812    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7b2e299-2b37-4d21-9264-1cbf9d15c274-istio-podinfo\") pod \"e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5\" (UID: \"f7b2e299-2b37-4d21-9264-1cbf9d15c274\") " pod="openshift-ingress/e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5"
Sep 14 10:04:23.798958 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:23.798866    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7b2e299-2b37-4d21-9264-1cbf9d15c274-istio-envoy\") pod \"e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5\" (UID: \"f7b2e299-2b37-4d21-9264-1cbf9d15c274\") " pod="openshift-ingress/e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5"
Sep 14 10:04:23.804946 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:23.804917    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7b2e299-2b37-4d21-9264-1cbf9d15c274-istio-token\") pod \"e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5\" (UID: \"f7b2e299-2b37-4d21-9264-1cbf9d15c274\") " pod="openshift-ingress/e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5"
Sep 14 10:04:23.805150 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:23.805129    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg87f\" (UniqueName: \"kubernetes.io/projected/f7b2e299-2b37-4d21-9264-1cbf9d15c274-kube-api-access-cg87f\") pod \"e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5\" (UID: \"f7b2e299-2b37-4d21-9264-1cbf9d15c274\") " pod="openshift-ingress/e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5"
Sep 14 10:04:23.870705 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:23.870661    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5"
Sep 14 10:04:23.970641 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:23.970603    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:24.042167 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:24.042131    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7678-fc9942" podUID="6bef203a-7ed2-491c-ba04-f6d58da7f06b" containerName="test-curl-7678-fc9942" containerID="cri-o://678274211bc9da16449f8cbfcdd0a75ffed894b7c7b89f546bfb78fc86d2d5a4" gracePeriod=2
Sep 14 10:04:24.171370 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:24.171336    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7678-03405c"
Sep 14 10:04:24.183812 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:24.183782    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5"]
Sep 14 10:04:24.184887 ip-10-0-132-4 kubenswrapper[2563]: W0914 10:04:24.184858    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7b2e299_2b37_4d21_9264_1cbf9d15c274.slice/crio-69b877abb2b543a923347c2669409407fe8f0668a785147c52e9be7819747a30 WatchSource:0}: Error finding container 69b877abb2b543a923347c2669409407fe8f0668a785147c52e9be7819747a30: Status 404 returned error can't find the container with id 69b877abb2b543a923347c2669409407fe8f0668a785147c52e9be7819747a30
Sep 14 10:04:24.187448 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:24.187413    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 10:04:24.187556 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:24.187485    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 10:04:24.187556 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:24.187511    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 10:04:24.301229 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:24.301114    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-frjk6\" (UniqueName: \"kubernetes.io/projected/8505d095-7937-4ae1-a3ca-0e1fb59b3049-kube-api-access-frjk6\") pod \"8505d095-7937-4ae1-a3ca-0e1fb59b3049\" (UID: \"8505d095-7937-4ae1-a3ca-0e1fb59b3049\") "
Sep 14 10:04:24.303458 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:24.303428    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8505d095-7937-4ae1-a3ca-0e1fb59b3049-kube-api-access-frjk6" (OuterVolumeSpecName: "kube-api-access-frjk6") pod "8505d095-7937-4ae1-a3ca-0e1fb59b3049" (UID: "8505d095-7937-4ae1-a3ca-0e1fb59b3049"). InnerVolumeSpecName "kube-api-access-frjk6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:04:24.403113 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:24.403067    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frjk6\" (UniqueName: \"kubernetes.io/projected/8505d095-7937-4ae1-a3ca-0e1fb59b3049-kube-api-access-frjk6\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:04:24.821477 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:24.821441    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:24.927692 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:24.927650    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:25.054296 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:25.054264    2563 generic.go:356] "Generic (PLEG): container finished" podID="8505d095-7937-4ae1-a3ca-0e1fb59b3049" containerID="071a414e7d75be885a4a06604dcc5ee1137cba04504877552a66261483b949de" exitCode=137
Sep 14 10:04:25.054719 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:25.054332    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7678-03405c"
Sep 14 10:04:25.054719 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:25.054365    2563 scope.go:117] "RemoveContainer" containerID="071a414e7d75be885a4a06604dcc5ee1137cba04504877552a66261483b949de"
Sep 14 10:04:25.056614 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:25.056530    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn"]
Sep 14 10:04:25.056800 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:25.056770    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn" podUID="a68cc340-1e5b-4971-96cc-91a62af7d642" containerName="istio-proxy" containerID="cri-o://6422193b01c0dfe2654946eb77ea4b11a6b868b997bb5244e1104c0a6b4f28e6" gracePeriod=30
Sep 14 10:04:25.057955 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:25.057350    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5" event={"ID":"f7b2e299-2b37-4d21-9264-1cbf9d15c274","Type":"ContainerStarted","Data":"bd56ee38bd9e54dada49c06a4a0c2cc9a307672c3314a785c10e1a1ad74e855b"}
Sep 14 10:04:25.057955 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:25.057419    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5" event={"ID":"f7b2e299-2b37-4d21-9264-1cbf9d15c274","Type":"ContainerStarted","Data":"69b877abb2b543a923347c2669409407fe8f0668a785147c52e9be7819747a30"}
Sep 14 10:04:25.067507 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:25.067481    2563 scope.go:117] "RemoveContainer" containerID="071a414e7d75be885a4a06604dcc5ee1137cba04504877552a66261483b949de"
Sep 14 10:04:25.067888 ip-10-0-132-4 kubenswrapper[2563]: E0914 10:04:25.067867    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"071a414e7d75be885a4a06604dcc5ee1137cba04504877552a66261483b949de\": container with ID starting with 071a414e7d75be885a4a06604dcc5ee1137cba04504877552a66261483b949de not found: ID does not exist" containerID="071a414e7d75be885a4a06604dcc5ee1137cba04504877552a66261483b949de"
Sep 14 10:04:25.067972 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:25.067895    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"071a414e7d75be885a4a06604dcc5ee1137cba04504877552a66261483b949de"} err="failed to get container status \"071a414e7d75be885a4a06604dcc5ee1137cba04504877552a66261483b949de\": rpc error: code = NotFound desc = could not find container \"071a414e7d75be885a4a06604dcc5ee1137cba04504877552a66261483b949de\": container with ID starting with 071a414e7d75be885a4a06604dcc5ee1137cba04504877552a66261483b949de not found: ID does not exist"
Sep 14 10:04:25.083587 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:25.083547    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:25.084118 ip-10-0-132-4 kubenswrapper[2563]: E0914 10:04:25.084093    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7678-03405c\" is forbidden: User \"system:node:ip-10-0-132-4.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-4.ec2.internal' and this object" podUID="8505d095-7937-4ae1-a3ca-0e1fb59b3049" pod="opendatahub/test-curl-7678-03405c"
Sep 14 10:04:25.105094 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:25.105034    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5" podStartSLOduration=2.105011161 podStartE2EDuration="2.105011161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:04:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:04:25.102512206 +0000 UTC m=+1254.142186838" watchObservedRunningTime="2026-09-14 10:04:25.105011161 +0000 UTC m=+1254.144685774"
Sep 14 10:04:25.415466 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:25.415362    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24873d-flxrt"]
Sep 14 10:04:25.418840 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:25.418810    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24873d-flxrt"
Sep 14 10:04:25.421297 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:25.421268    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gf4xj\""
Sep 14 10:04:25.421458 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:25.421270    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 14 10:04:25.428614 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:25.428579    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24873d-flxrt"]
Sep 14 10:04:25.440431 ip-10-0-132-4 kubenswrapper[2563]: E0914 10:04:25.440386    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7678-03405c\" is forbidden: User \"system:node:ip-10-0-132-4.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-4.ec2.internal' and this object" podUID="8505d095-7937-4ae1-a3ca-0e1fb59b3049" pod="opendatahub/test-curl-7678-03405c"
Sep 14 10:04:25.483988 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:25.483947    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8505d095-7937-4ae1-a3ca-0e1fb59b3049" path="/var/lib/kubelet/pods/8505d095-7937-4ae1-a3ca-0e1fb59b3049/volumes"
Sep 14 10:04:25.517374 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:25.517330    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a42d38e4-1fe2-4156-bc00-92bc19bcedbd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-24873d-flxrt\" (UID: \"a42d38e4-1fe2-4156-bc00-92bc19bcedbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24873d-flxrt"
Sep 14 10:04:25.562503 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:25.562461    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:25.623938 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:25.618607    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a42d38e4-1fe2-4156-bc00-92bc19bcedbd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-24873d-flxrt\" (UID: \"a42d38e4-1fe2-4156-bc00-92bc19bcedbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24873d-flxrt"
Sep 14 10:04:25.623938 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:25.619632    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a42d38e4-1fe2-4156-bc00-92bc19bcedbd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-24873d-flxrt\" (UID: \"a42d38e4-1fe2-4156-bc00-92bc19bcedbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24873d-flxrt"
Sep 14 10:04:25.646888 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:25.646850    2563 patch_prober.go:28] interesting pod/e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 14 10:04:25.647050 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:25.646914    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn" podUID="a68cc340-1e5b-4971-96cc-91a62af7d642" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 14 10:04:25.732478 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:25.732377    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24873d-flxrt"
Sep 14 10:04:25.871857 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:25.871815    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5"
Sep 14 10:04:25.877486 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:25.877456    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5"
Sep 14 10:04:26.007646 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:26.007613    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24873d-flxrt"]
Sep 14 10:04:26.009206 ip-10-0-132-4 kubenswrapper[2563]: W0914 10:04:26.009176    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda42d38e4_1fe2_4156_bc00_92bc19bcedbd.slice/crio-8eb3b320ed59d0b70e4a7c5a4c0e98402cc0eed50ab6bae6aa80385093146eb9 WatchSource:0}: Error finding container 8eb3b320ed59d0b70e4a7c5a4c0e98402cc0eed50ab6bae6aa80385093146eb9: Status 404 returned error can't find the container with id 8eb3b320ed59d0b70e4a7c5a4c0e98402cc0eed50ab6bae6aa80385093146eb9
Sep 14 10:04:26.066211 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:26.066169    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24873d-flxrt" event={"ID":"a42d38e4-1fe2-4156-bc00-92bc19bcedbd","Type":"ContainerStarted","Data":"8eb3b320ed59d0b70e4a7c5a4c0e98402cc0eed50ab6bae6aa80385093146eb9"}
Sep 14 10:04:26.067619 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:26.067590    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5"
Sep 14 10:04:26.068656 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:26.068632    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5"
Sep 14 10:04:26.255974 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:26.255885    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7678-fc9942"
Sep 14 10:04:26.326787 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:26.326755    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65tnr\" (UniqueName: \"kubernetes.io/projected/6bef203a-7ed2-491c-ba04-f6d58da7f06b-kube-api-access-65tnr\") pod \"6bef203a-7ed2-491c-ba04-f6d58da7f06b\" (UID: \"6bef203a-7ed2-491c-ba04-f6d58da7f06b\") "
Sep 14 10:04:26.328944 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:26.328919    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bef203a-7ed2-491c-ba04-f6d58da7f06b-kube-api-access-65tnr" (OuterVolumeSpecName: "kube-api-access-65tnr") pod "6bef203a-7ed2-491c-ba04-f6d58da7f06b" (UID: "6bef203a-7ed2-491c-ba04-f6d58da7f06b"). InnerVolumeSpecName "kube-api-access-65tnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:04:26.376835 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:26.376796    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-24873d/test-model-24873d-kserve-dff86779c-29lrm" podUID="0d55a291-49c9-4c1e-8ae1-7cf6632d4395" containerName="main" containerID="cri-o://d3f5faa3bb17859a0b83bb52a0cf6346c61bf5e3b2b752b1e70c2c9d61c920ed" gracePeriod=45
Sep 14 10:04:26.428449 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:26.428404    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65tnr\" (UniqueName: \"kubernetes.io/projected/6bef203a-7ed2-491c-ba04-f6d58da7f06b-kube-api-access-65tnr\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:04:26.546939 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:26.546837    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:26.632797 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:26.632766    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-24873d/test-model-24873d-kserve-dff86779c-29lrm"
Sep 14 10:04:26.731205 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:26.731110    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d55a291-49c9-4c1e-8ae1-7cf6632d4395-tmp-dir\") pod \"0d55a291-49c9-4c1e-8ae1-7cf6632d4395\" (UID: \"0d55a291-49c9-4c1e-8ae1-7cf6632d4395\") "
Sep 14 10:04:26.731205 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:26.731203    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d55a291-49c9-4c1e-8ae1-7cf6632d4395-model-cache\") pod \"0d55a291-49c9-4c1e-8ae1-7cf6632d4395\" (UID: \"0d55a291-49c9-4c1e-8ae1-7cf6632d4395\") "
Sep 14 10:04:26.731328 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:26.731308    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gx88v\" (UniqueName: \"kubernetes.io/projected/0d55a291-49c9-4c1e-8ae1-7cf6632d4395-kube-api-access-gx88v\") pod \"0d55a291-49c9-4c1e-8ae1-7cf6632d4395\" (UID: \"0d55a291-49c9-4c1e-8ae1-7cf6632d4395\") "
Sep 14 10:04:26.731492 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:26.731450    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d55a291-49c9-4c1e-8ae1-7cf6632d4395-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0d55a291-49c9-4c1e-8ae1-7cf6632d4395" (UID: "0d55a291-49c9-4c1e-8ae1-7cf6632d4395"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:04:26.731670 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:26.731499    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d55a291-49c9-4c1e-8ae1-7cf6632d4395-model-cache" (OuterVolumeSpecName: "model-cache") pod "0d55a291-49c9-4c1e-8ae1-7cf6632d4395" (UID: "0d55a291-49c9-4c1e-8ae1-7cf6632d4395"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:04:26.731670 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:26.731587    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d55a291-49c9-4c1e-8ae1-7cf6632d4395-home\") pod \"0d55a291-49c9-4c1e-8ae1-7cf6632d4395\" (UID: \"0d55a291-49c9-4c1e-8ae1-7cf6632d4395\") "
Sep 14 10:04:26.731670 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:26.731647    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d55a291-49c9-4c1e-8ae1-7cf6632d4395-dshm\") pod \"0d55a291-49c9-4c1e-8ae1-7cf6632d4395\" (UID: \"0d55a291-49c9-4c1e-8ae1-7cf6632d4395\") "
Sep 14 10:04:26.731833 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:26.731722    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d55a291-49c9-4c1e-8ae1-7cf6632d4395-tls-certs\") pod \"0d55a291-49c9-4c1e-8ae1-7cf6632d4395\" (UID: \"0d55a291-49c9-4c1e-8ae1-7cf6632d4395\") "
Sep 14 10:04:26.731833 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:26.731731    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d55a291-49c9-4c1e-8ae1-7cf6632d4395-home" (OuterVolumeSpecName: "home") pod "0d55a291-49c9-4c1e-8ae1-7cf6632d4395" (UID: "0d55a291-49c9-4c1e-8ae1-7cf6632d4395"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:04:26.732162 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:26.732131    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d55a291-49c9-4c1e-8ae1-7cf6632d4395-tmp-dir\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:04:26.732162 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:26.732154    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d55a291-49c9-4c1e-8ae1-7cf6632d4395-model-cache\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:04:26.732320 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:26.732167    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d55a291-49c9-4c1e-8ae1-7cf6632d4395-home\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:04:26.733950 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:26.733922    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d55a291-49c9-4c1e-8ae1-7cf6632d4395-dshm" (OuterVolumeSpecName: "dshm") pod "0d55a291-49c9-4c1e-8ae1-7cf6632d4395" (UID: "0d55a291-49c9-4c1e-8ae1-7cf6632d4395"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:04:26.734088 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:26.733941    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d55a291-49c9-4c1e-8ae1-7cf6632d4395-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0d55a291-49c9-4c1e-8ae1-7cf6632d4395" (UID: "0d55a291-49c9-4c1e-8ae1-7cf6632d4395"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:04:26.734088 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:26.733961    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d55a291-49c9-4c1e-8ae1-7cf6632d4395-kube-api-access-gx88v" (OuterVolumeSpecName: "kube-api-access-gx88v") pod "0d55a291-49c9-4c1e-8ae1-7cf6632d4395" (UID: "0d55a291-49c9-4c1e-8ae1-7cf6632d4395"). InnerVolumeSpecName "kube-api-access-gx88v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:04:26.833089 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:26.832954    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx88v\" (UniqueName: \"kubernetes.io/projected/0d55a291-49c9-4c1e-8ae1-7cf6632d4395-kube-api-access-gx88v\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:04:26.833089 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:26.832988    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d55a291-49c9-4c1e-8ae1-7cf6632d4395-dshm\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:04:26.833089 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:26.833029    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d55a291-49c9-4c1e-8ae1-7cf6632d4395-tls-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:04:26.978353 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:26.978310    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-2901ec/e2e-conflict-b-c109e646-kserve-7d9d8884d9-b6x76" podUID="53af90b8-85ec-4fa4-9831-59e5e6ce0037" containerName="main" containerID="cri-o://e0c4f16fd7e7d45394a872be9a23d0e081f2f315507597067f5787f0a45285c3" gracePeriod=45
Sep 14 10:04:27.056353 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:27.056318    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:27.076092 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:27.076040    2563 generic.go:356] "Generic (PLEG): container finished" podID="6bef203a-7ed2-491c-ba04-f6d58da7f06b" containerID="678274211bc9da16449f8cbfcdd0a75ffed894b7c7b89f546bfb78fc86d2d5a4" exitCode=137
Sep 14 10:04:27.076532 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:27.076139    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7678-fc9942"
Sep 14 10:04:27.076532 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:27.076179    2563 scope.go:117] "RemoveContainer" containerID="678274211bc9da16449f8cbfcdd0a75ffed894b7c7b89f546bfb78fc86d2d5a4"
Sep 14 10:04:27.077949 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:27.077927    2563 generic.go:356] "Generic (PLEG): container finished" podID="0d55a291-49c9-4c1e-8ae1-7cf6632d4395" containerID="d3f5faa3bb17859a0b83bb52a0cf6346c61bf5e3b2b752b1e70c2c9d61c920ed" exitCode=0
Sep 14 10:04:27.078091 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:27.077972    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-24873d/test-model-24873d-kserve-dff86779c-29lrm"
Sep 14 10:04:27.078091 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:27.078022    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-24873d/test-model-24873d-kserve-dff86779c-29lrm" event={"ID":"0d55a291-49c9-4c1e-8ae1-7cf6632d4395","Type":"ContainerDied","Data":"d3f5faa3bb17859a0b83bb52a0cf6346c61bf5e3b2b752b1e70c2c9d61c920ed"}
Sep 14 10:04:27.078091 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:27.078068    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-24873d/test-model-24873d-kserve-dff86779c-29lrm" event={"ID":"0d55a291-49c9-4c1e-8ae1-7cf6632d4395","Type":"ContainerDied","Data":"eb883d87c4df91e19077fd77e4aa57cd91ce02dd902ddc94906c86948da35e44"}
Sep 14 10:04:27.080386 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:27.080350    2563 generic.go:356] "Generic (PLEG): container finished" podID="a42d38e4-1fe2-4156-bc00-92bc19bcedbd" containerID="e15a95802a047439f8f88fba14f5551479206b8edb85be9081cd8bedae86b378" exitCode=0
Sep 14 10:04:27.080991 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:27.080957    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24873d-flxrt" event={"ID":"a42d38e4-1fe2-4156-bc00-92bc19bcedbd","Type":"ContainerDied","Data":"e15a95802a047439f8f88fba14f5551479206b8edb85be9081cd8bedae86b378"}
Sep 14 10:04:27.093193 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:27.093039    2563 scope.go:117] "RemoveContainer" containerID="678274211bc9da16449f8cbfcdd0a75ffed894b7c7b89f546bfb78fc86d2d5a4"
Sep 14 10:04:27.093454 ip-10-0-132-4 kubenswrapper[2563]: E0914 10:04:27.093432    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"678274211bc9da16449f8cbfcdd0a75ffed894b7c7b89f546bfb78fc86d2d5a4\": container with ID starting with 678274211bc9da16449f8cbfcdd0a75ffed894b7c7b89f546bfb78fc86d2d5a4 not found: ID does not exist" containerID="678274211bc9da16449f8cbfcdd0a75ffed894b7c7b89f546bfb78fc86d2d5a4"
Sep 14 10:04:27.093508 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:27.093465    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"678274211bc9da16449f8cbfcdd0a75ffed894b7c7b89f546bfb78fc86d2d5a4"} err="failed to get container status \"678274211bc9da16449f8cbfcdd0a75ffed894b7c7b89f546bfb78fc86d2d5a4\": rpc error: code = NotFound desc = could not find container \"678274211bc9da16449f8cbfcdd0a75ffed894b7c7b89f546bfb78fc86d2d5a4\": container with ID starting with 678274211bc9da16449f8cbfcdd0a75ffed894b7c7b89f546bfb78fc86d2d5a4 not found: ID does not exist"
Sep 14 10:04:27.093508 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:27.093486    2563 scope.go:117] "RemoveContainer" containerID="d3f5faa3bb17859a0b83bb52a0cf6346c61bf5e3b2b752b1e70c2c9d61c920ed"
Sep 14 10:04:27.104169 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:27.104137    2563 scope.go:117] "RemoveContainer" containerID="d3f5faa3bb17859a0b83bb52a0cf6346c61bf5e3b2b752b1e70c2c9d61c920ed"
Sep 14 10:04:27.104599 ip-10-0-132-4 kubenswrapper[2563]: E0914 10:04:27.104563    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3f5faa3bb17859a0b83bb52a0cf6346c61bf5e3b2b752b1e70c2c9d61c920ed\": container with ID starting with d3f5faa3bb17859a0b83bb52a0cf6346c61bf5e3b2b752b1e70c2c9d61c920ed not found: ID does not exist" containerID="d3f5faa3bb17859a0b83bb52a0cf6346c61bf5e3b2b752b1e70c2c9d61c920ed"
Sep 14 10:04:27.104712 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:27.104606    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3f5faa3bb17859a0b83bb52a0cf6346c61bf5e3b2b752b1e70c2c9d61c920ed"} err="failed to get container status \"d3f5faa3bb17859a0b83bb52a0cf6346c61bf5e3b2b752b1e70c2c9d61c920ed\": rpc error: code = NotFound desc = could not find container \"d3f5faa3bb17859a0b83bb52a0cf6346c61bf5e3b2b752b1e70c2c9d61c920ed\": container with ID starting with d3f5faa3bb17859a0b83bb52a0cf6346c61bf5e3b2b752b1e70c2c9d61c920ed not found: ID does not exist"
Sep 14 10:04:27.120931 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:27.120882    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-24873d/test-model-24873d-kserve-dff86779c-29lrm"]
Sep 14 10:04:27.123232 ip-10-0-132-4 kubenswrapper[2563]: E0914 10:04:27.123190    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7678-fc9942\" is forbidden: User \"system:node:ip-10-0-132-4.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-4.ec2.internal' and this object" podUID="6bef203a-7ed2-491c-ba04-f6d58da7f06b" pod="opendatahub/test-curl-7678-fc9942"
Sep 14 10:04:27.125836 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:27.125800    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-24873d/test-model-24873d-kserve-dff86779c-29lrm"]
Sep 14 10:04:27.146282 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:27.146252    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:27.248071 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:27.248024    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2901ec/e2e-conflict-b-c109e646-kserve-7d9d8884d9-b6x76"
Sep 14 10:04:27.250204 ip-10-0-132-4 kubenswrapper[2563]: E0914 10:04:27.250173    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7678-fc9942\" is forbidden: User \"system:node:ip-10-0-132-4.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-4.ec2.internal' and this object" podUID="6bef203a-7ed2-491c-ba04-f6d58da7f06b" pod="opendatahub/test-curl-7678-fc9942"
Sep 14 10:04:27.339109 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:27.338976    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53af90b8-85ec-4fa4-9831-59e5e6ce0037-tls-certs\") pod \"53af90b8-85ec-4fa4-9831-59e5e6ce0037\" (UID: \"53af90b8-85ec-4fa4-9831-59e5e6ce0037\") "
Sep 14 10:04:27.339109 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:27.339098    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53af90b8-85ec-4fa4-9831-59e5e6ce0037-dshm\") pod \"53af90b8-85ec-4fa4-9831-59e5e6ce0037\" (UID: \"53af90b8-85ec-4fa4-9831-59e5e6ce0037\") "
Sep 14 10:04:27.339313 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:27.339213    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53af90b8-85ec-4fa4-9831-59e5e6ce0037-model-cache\") pod \"53af90b8-85ec-4fa4-9831-59e5e6ce0037\" (UID: \"53af90b8-85ec-4fa4-9831-59e5e6ce0037\") "
Sep 14 10:04:27.339313 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:27.339270    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8tb9t\" (UniqueName: \"kubernetes.io/projected/53af90b8-85ec-4fa4-9831-59e5e6ce0037-kube-api-access-8tb9t\") pod \"53af90b8-85ec-4fa4-9831-59e5e6ce0037\" (UID: \"53af90b8-85ec-4fa4-9831-59e5e6ce0037\") "
Sep 14 10:04:27.339391 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:27.339320    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53af90b8-85ec-4fa4-9831-59e5e6ce0037-home\") pod \"53af90b8-85ec-4fa4-9831-59e5e6ce0037\" (UID: \"53af90b8-85ec-4fa4-9831-59e5e6ce0037\") "
Sep 14 10:04:27.339391 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:27.339351    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53af90b8-85ec-4fa4-9831-59e5e6ce0037-tmp-dir\") pod \"53af90b8-85ec-4fa4-9831-59e5e6ce0037\" (UID: \"53af90b8-85ec-4fa4-9831-59e5e6ce0037\") "
Sep 14 10:04:27.339540 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:27.339505    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53af90b8-85ec-4fa4-9831-59e5e6ce0037-model-cache" (OuterVolumeSpecName: "model-cache") pod "53af90b8-85ec-4fa4-9831-59e5e6ce0037" (UID: "53af90b8-85ec-4fa4-9831-59e5e6ce0037"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:04:27.339608 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:27.339545    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53af90b8-85ec-4fa4-9831-59e5e6ce0037-home" (OuterVolumeSpecName: "home") pod "53af90b8-85ec-4fa4-9831-59e5e6ce0037" (UID: "53af90b8-85ec-4fa4-9831-59e5e6ce0037"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:04:27.339719 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:27.339683    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53af90b8-85ec-4fa4-9831-59e5e6ce0037-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "53af90b8-85ec-4fa4-9831-59e5e6ce0037" (UID: "53af90b8-85ec-4fa4-9831-59e5e6ce0037"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:04:27.339875 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:27.339859    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53af90b8-85ec-4fa4-9831-59e5e6ce0037-model-cache\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:04:27.339938 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:27.339879    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53af90b8-85ec-4fa4-9831-59e5e6ce0037-home\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:04:27.339938 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:27.339893    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53af90b8-85ec-4fa4-9831-59e5e6ce0037-tmp-dir\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:04:27.341544 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:27.341507    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53af90b8-85ec-4fa4-9831-59e5e6ce0037-dshm" (OuterVolumeSpecName: "dshm") pod "53af90b8-85ec-4fa4-9831-59e5e6ce0037" (UID: "53af90b8-85ec-4fa4-9831-59e5e6ce0037"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:04:27.341544 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:27.341527    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53af90b8-85ec-4fa4-9831-59e5e6ce0037-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "53af90b8-85ec-4fa4-9831-59e5e6ce0037" (UID: "53af90b8-85ec-4fa4-9831-59e5e6ce0037"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:04:27.341700 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:27.341573    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53af90b8-85ec-4fa4-9831-59e5e6ce0037-kube-api-access-8tb9t" (OuterVolumeSpecName: "kube-api-access-8tb9t") pod "53af90b8-85ec-4fa4-9831-59e5e6ce0037" (UID: "53af90b8-85ec-4fa4-9831-59e5e6ce0037"). InnerVolumeSpecName "kube-api-access-8tb9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:04:27.440792 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:27.440752    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53af90b8-85ec-4fa4-9831-59e5e6ce0037-dshm\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:04:27.440792 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:27.440786    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tb9t\" (UniqueName: \"kubernetes.io/projected/53af90b8-85ec-4fa4-9831-59e5e6ce0037-kube-api-access-8tb9t\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:04:27.440792 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:27.440799    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53af90b8-85ec-4fa4-9831-59e5e6ce0037-tls-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:04:27.484613 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:27.484567    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d55a291-49c9-4c1e-8ae1-7cf6632d4395" path="/var/lib/kubelet/pods/0d55a291-49c9-4c1e-8ae1-7cf6632d4395/volumes"
Sep 14 10:04:27.484964 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:27.484950    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bef203a-7ed2-491c-ba04-f6d58da7f06b" path="/var/lib/kubelet/pods/6bef203a-7ed2-491c-ba04-f6d58da7f06b/volumes"
Sep 14 10:04:27.730567 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:27.730476    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:28.095468 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:28.095432    2563 generic.go:356] "Generic (PLEG): container finished" podID="53af90b8-85ec-4fa4-9831-59e5e6ce0037" containerID="e0c4f16fd7e7d45394a872be9a23d0e081f2f315507597067f5787f0a45285c3" exitCode=0
Sep 14 10:04:28.095921 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:28.095504    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2901ec/e2e-conflict-b-c109e646-kserve-7d9d8884d9-b6x76"
Sep 14 10:04:28.095921 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:28.095522    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2901ec/e2e-conflict-b-c109e646-kserve-7d9d8884d9-b6x76" event={"ID":"53af90b8-85ec-4fa4-9831-59e5e6ce0037","Type":"ContainerDied","Data":"e0c4f16fd7e7d45394a872be9a23d0e081f2f315507597067f5787f0a45285c3"}
Sep 14 10:04:28.095921 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:28.095570    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2901ec/e2e-conflict-b-c109e646-kserve-7d9d8884d9-b6x76" event={"ID":"53af90b8-85ec-4fa4-9831-59e5e6ce0037","Type":"ContainerDied","Data":"f654e1a8e4b95fe7495f1016ae0c1907854b13a3d22045ac78f3023c2155e839"}
Sep 14 10:04:28.095921 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:28.095594    2563 scope.go:117] "RemoveContainer" containerID="e0c4f16fd7e7d45394a872be9a23d0e081f2f315507597067f5787f0a45285c3"
Sep 14 10:04:28.106739 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:28.106711    2563 scope.go:117] "RemoveContainer" containerID="e0c4f16fd7e7d45394a872be9a23d0e081f2f315507597067f5787f0a45285c3"
Sep 14 10:04:28.107159 ip-10-0-132-4 kubenswrapper[2563]: E0914 10:04:28.107138    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0c4f16fd7e7d45394a872be9a23d0e081f2f315507597067f5787f0a45285c3\": container with ID starting with e0c4f16fd7e7d45394a872be9a23d0e081f2f315507597067f5787f0a45285c3 not found: ID does not exist" containerID="e0c4f16fd7e7d45394a872be9a23d0e081f2f315507597067f5787f0a45285c3"
Sep 14 10:04:28.107242 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:28.107170    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0c4f16fd7e7d45394a872be9a23d0e081f2f315507597067f5787f0a45285c3"} err="failed to get container status \"e0c4f16fd7e7d45394a872be9a23d0e081f2f315507597067f5787f0a45285c3\": rpc error: code = NotFound desc = could not find container \"e0c4f16fd7e7d45394a872be9a23d0e081f2f315507597067f5787f0a45285c3\": container with ID starting with e0c4f16fd7e7d45394a872be9a23d0e081f2f315507597067f5787f0a45285c3 not found: ID does not exist"
Sep 14 10:04:28.117804 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:28.117752    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-2901ec/e2e-conflict-b-c109e646-kserve-7d9d8884d9-b6x76"]
Sep 14 10:04:28.122508 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:28.122470    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-2901ec/e2e-conflict-b-c109e646-kserve-7d9d8884d9-b6x76"]
Sep 14 10:04:28.249567 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:28.249532    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24873d-flxrt"
Sep 14 10:04:28.350868 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:28.350760    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a42d38e4-1fe2-4156-bc00-92bc19bcedbd-maas-api-service-ca\") pod \"a42d38e4-1fe2-4156-bc00-92bc19bcedbd\" (UID: \"a42d38e4-1fe2-4156-bc00-92bc19bcedbd\") "
Sep 14 10:04:28.351178 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:28.351154    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a42d38e4-1fe2-4156-bc00-92bc19bcedbd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a42d38e4-1fe2-4156-bc00-92bc19bcedbd" (UID: "a42d38e4-1fe2-4156-bc00-92bc19bcedbd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:04:28.452386 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:28.452340    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a42d38e4-1fe2-4156-bc00-92bc19bcedbd-maas-api-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:04:28.686720 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:28.686629    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:29.106742 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:29.106707    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24873d-flxrt"
Sep 14 10:04:29.107222 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:29.106743    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24873d-flxrt" event={"ID":"a42d38e4-1fe2-4156-bc00-92bc19bcedbd","Type":"ContainerDied","Data":"8eb3b320ed59d0b70e4a7c5a4c0e98402cc0eed50ab6bae6aa80385093146eb9"}
Sep 14 10:04:29.107222 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:29.106792    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8eb3b320ed59d0b70e4a7c5a4c0e98402cc0eed50ab6bae6aa80385093146eb9"
Sep 14 10:04:29.483214 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:29.483110    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53af90b8-85ec-4fa4-9831-59e5e6ce0037" path="/var/lib/kubelet/pods/53af90b8-85ec-4fa4-9831-59e5e6ce0037/volumes"
Sep 14 10:04:29.493491 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:29.493460    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:30.276399 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:30.276366    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:30.325966 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:30.325890    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn"
Sep 14 10:04:30.473578 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:30.473535    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m4vbs\" (UniqueName: \"kubernetes.io/projected/a68cc340-1e5b-4971-96cc-91a62af7d642-kube-api-access-m4vbs\") pod \"a68cc340-1e5b-4971-96cc-91a62af7d642\" (UID: \"a68cc340-1e5b-4971-96cc-91a62af7d642\") "
Sep 14 10:04:30.473765 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:30.473589    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a68cc340-1e5b-4971-96cc-91a62af7d642-workload-certs\") pod \"a68cc340-1e5b-4971-96cc-91a62af7d642\" (UID: \"a68cc340-1e5b-4971-96cc-91a62af7d642\") "
Sep 14 10:04:30.473765 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:30.473660    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a68cc340-1e5b-4971-96cc-91a62af7d642-istio-podinfo\") pod \"a68cc340-1e5b-4971-96cc-91a62af7d642\" (UID: \"a68cc340-1e5b-4971-96cc-91a62af7d642\") "
Sep 14 10:04:30.473765 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:30.473703    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a68cc340-1e5b-4971-96cc-91a62af7d642-istio-token\") pod \"a68cc340-1e5b-4971-96cc-91a62af7d642\" (UID: \"a68cc340-1e5b-4971-96cc-91a62af7d642\") "
Sep 14 10:04:30.473765 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:30.473739    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a68cc340-1e5b-4971-96cc-91a62af7d642-istiod-ca-cert\") pod \"a68cc340-1e5b-4971-96cc-91a62af7d642\" (UID: \"a68cc340-1e5b-4971-96cc-91a62af7d642\") "
Sep 14 10:04:30.473966 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:30.473797    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a68cc340-1e5b-4971-96cc-91a62af7d642-istio-envoy\") pod \"a68cc340-1e5b-4971-96cc-91a62af7d642\" (UID: \"a68cc340-1e5b-4971-96cc-91a62af7d642\") "
Sep 14 10:04:30.473966 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:30.473834    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a68cc340-1e5b-4971-96cc-91a62af7d642-workload-socket\") pod \"a68cc340-1e5b-4971-96cc-91a62af7d642\" (UID: \"a68cc340-1e5b-4971-96cc-91a62af7d642\") "
Sep 14 10:04:30.473966 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:30.473859    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a68cc340-1e5b-4971-96cc-91a62af7d642-istio-data\") pod \"a68cc340-1e5b-4971-96cc-91a62af7d642\" (UID: \"a68cc340-1e5b-4971-96cc-91a62af7d642\") "
Sep 14 10:04:30.473966 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:30.473892    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a68cc340-1e5b-4971-96cc-91a62af7d642-credential-socket\") pod \"a68cc340-1e5b-4971-96cc-91a62af7d642\" (UID: \"a68cc340-1e5b-4971-96cc-91a62af7d642\") "
Sep 14 10:04:30.473966 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:30.473942    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a68cc340-1e5b-4971-96cc-91a62af7d642-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a68cc340-1e5b-4971-96cc-91a62af7d642" (UID: "a68cc340-1e5b-4971-96cc-91a62af7d642"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:04:30.474262 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:30.474154    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a68cc340-1e5b-4971-96cc-91a62af7d642-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a68cc340-1e5b-4971-96cc-91a62af7d642" (UID: "a68cc340-1e5b-4971-96cc-91a62af7d642"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:04:30.474320 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:30.474255    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a68cc340-1e5b-4971-96cc-91a62af7d642-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a68cc340-1e5b-4971-96cc-91a62af7d642" (UID: "a68cc340-1e5b-4971-96cc-91a62af7d642"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:04:30.474320 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:30.474264    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a68cc340-1e5b-4971-96cc-91a62af7d642-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a68cc340-1e5b-4971-96cc-91a62af7d642" (UID: "a68cc340-1e5b-4971-96cc-91a62af7d642"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:04:30.474417 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:30.474380    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a68cc340-1e5b-4971-96cc-91a62af7d642-istio-data" (OuterVolumeSpecName: "istio-data") pod "a68cc340-1e5b-4971-96cc-91a62af7d642" (UID: "a68cc340-1e5b-4971-96cc-91a62af7d642"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:04:30.474474 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:30.474431    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a68cc340-1e5b-4971-96cc-91a62af7d642-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:04:30.474474 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:30.474452    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a68cc340-1e5b-4971-96cc-91a62af7d642-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:04:30.474474 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:30.474465    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a68cc340-1e5b-4971-96cc-91a62af7d642-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:04:30.474633 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:30.474477    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a68cc340-1e5b-4971-96cc-91a62af7d642-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:04:30.476372 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:30.476328    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a68cc340-1e5b-4971-96cc-91a62af7d642-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a68cc340-1e5b-4971-96cc-91a62af7d642" (UID: "a68cc340-1e5b-4971-96cc-91a62af7d642"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 10:04:30.476518 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:30.476470    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a68cc340-1e5b-4971-96cc-91a62af7d642-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a68cc340-1e5b-4971-96cc-91a62af7d642" (UID: "a68cc340-1e5b-4971-96cc-91a62af7d642"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:04:30.476518 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:30.476478    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a68cc340-1e5b-4971-96cc-91a62af7d642-kube-api-access-m4vbs" (OuterVolumeSpecName: "kube-api-access-m4vbs") pod "a68cc340-1e5b-4971-96cc-91a62af7d642" (UID: "a68cc340-1e5b-4971-96cc-91a62af7d642"). InnerVolumeSpecName "kube-api-access-m4vbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:04:30.476518 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:30.476484    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a68cc340-1e5b-4971-96cc-91a62af7d642-istio-token" (OuterVolumeSpecName: "istio-token") pod "a68cc340-1e5b-4971-96cc-91a62af7d642" (UID: "a68cc340-1e5b-4971-96cc-91a62af7d642"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:04:30.575312 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:30.575266    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a68cc340-1e5b-4971-96cc-91a62af7d642-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:04:30.575312 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:30.575300    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a68cc340-1e5b-4971-96cc-91a62af7d642-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:04:30.575312 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:30.575309    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a68cc340-1e5b-4971-96cc-91a62af7d642-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:04:30.575312 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:30.575317    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a68cc340-1e5b-4971-96cc-91a62af7d642-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:04:30.575312 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:30.575325    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4vbs\" (UniqueName: \"kubernetes.io/projected/a68cc340-1e5b-4971-96cc-91a62af7d642-kube-api-access-m4vbs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:04:31.122959 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:31.122919    2563 generic.go:356] "Generic (PLEG): container finished" podID="a68cc340-1e5b-4971-96cc-91a62af7d642" containerID="6422193b01c0dfe2654946eb77ea4b11a6b868b997bb5244e1104c0a6b4f28e6" exitCode=0
Sep 14 10:04:31.123168 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:31.122972    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn"
Sep 14 10:04:31.123168 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:31.123029    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn" event={"ID":"a68cc340-1e5b-4971-96cc-91a62af7d642","Type":"ContainerDied","Data":"6422193b01c0dfe2654946eb77ea4b11a6b868b997bb5244e1104c0a6b4f28e6"}
Sep 14 10:04:31.123168 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:31.123087    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn" event={"ID":"a68cc340-1e5b-4971-96cc-91a62af7d642","Type":"ContainerDied","Data":"6cfa0b66e2deb2384a8f7b249708dec1f40ecfd6f7a8e80dea0f227eba907eb3"}
Sep 14 10:04:31.123168 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:31.123110    2563 scope.go:117] "RemoveContainer" containerID="6422193b01c0dfe2654946eb77ea4b11a6b868b997bb5244e1104c0a6b4f28e6"
Sep 14 10:04:31.133690 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:31.133666    2563 scope.go:117] "RemoveContainer" containerID="2582027d20843d4e4febcf69ccbb6c0043ba638676372fb2b834cfa4cea62f31"
Sep 14 10:04:31.144156 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:31.144134    2563 scope.go:117] "RemoveContainer" containerID="6422193b01c0dfe2654946eb77ea4b11a6b868b997bb5244e1104c0a6b4f28e6"
Sep 14 10:04:31.144515 ip-10-0-132-4 kubenswrapper[2563]: E0914 10:04:31.144494    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6422193b01c0dfe2654946eb77ea4b11a6b868b997bb5244e1104c0a6b4f28e6\": container with ID starting with 6422193b01c0dfe2654946eb77ea4b11a6b868b997bb5244e1104c0a6b4f28e6 not found: ID does not exist" containerID="6422193b01c0dfe2654946eb77ea4b11a6b868b997bb5244e1104c0a6b4f28e6"
Sep 14 10:04:31.144599 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:31.144531    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6422193b01c0dfe2654946eb77ea4b11a6b868b997bb5244e1104c0a6b4f28e6"} err="failed to get container status \"6422193b01c0dfe2654946eb77ea4b11a6b868b997bb5244e1104c0a6b4f28e6\": rpc error: code = NotFound desc = could not find container \"6422193b01c0dfe2654946eb77ea4b11a6b868b997bb5244e1104c0a6b4f28e6\": container with ID starting with 6422193b01c0dfe2654946eb77ea4b11a6b868b997bb5244e1104c0a6b4f28e6 not found: ID does not exist"
Sep 14 10:04:31.144599 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:31.144559    2563 scope.go:117] "RemoveContainer" containerID="2582027d20843d4e4febcf69ccbb6c0043ba638676372fb2b834cfa4cea62f31"
Sep 14 10:04:31.144898 ip-10-0-132-4 kubenswrapper[2563]: E0914 10:04:31.144880    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2582027d20843d4e4febcf69ccbb6c0043ba638676372fb2b834cfa4cea62f31\": container with ID starting with 2582027d20843d4e4febcf69ccbb6c0043ba638676372fb2b834cfa4cea62f31 not found: ID does not exist" containerID="2582027d20843d4e4febcf69ccbb6c0043ba638676372fb2b834cfa4cea62f31"
Sep 14 10:04:31.144939 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:31.144905    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2582027d20843d4e4febcf69ccbb6c0043ba638676372fb2b834cfa4cea62f31"} err="failed to get container status \"2582027d20843d4e4febcf69ccbb6c0043ba638676372fb2b834cfa4cea62f31\": rpc error: code = NotFound desc = could not find container \"2582027d20843d4e4febcf69ccbb6c0043ba638676372fb2b834cfa4cea62f31\": container with ID starting with 2582027d20843d4e4febcf69ccbb6c0043ba638676372fb2b834cfa4cea62f31 not found: ID does not exist"
Sep 14 10:04:31.152111 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:31.152014    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn"]
Sep 14 10:04:31.165794 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:31.165756    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-0e60f3-openshift-default-68dd4bdb98-7jwcn"]
Sep 14 10:04:31.203407 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:31.203374    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:31.283081 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:31.283046    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:31.483543 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:31.483452    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a68cc340-1e5b-4971-96cc-91a62af7d642" path="/var/lib/kubelet/pods/a68cc340-1e5b-4971-96cc-91a62af7d642/volumes"
Sep 14 10:04:31.566810 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:31.566776    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:31.593395 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:31.593355    2563 scope.go:117] "RemoveContainer" containerID="ec15fb459157c21ab16242fd71d326a98dc301971b7c7e09b3aee9f284c297e0"
Sep 14 10:04:31.605979 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:31.605951    2563 scope.go:117] "RemoveContainer" containerID="a42dcf3ce099ff2285c25a2a66947847af7b7823ef5c9bc0e7d6b0dde51e87df"
Sep 14 10:04:31.700330 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:31.700288    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:31.800750 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:31.800720    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:32.147554 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:32.147454    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:33.213617 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:33.213582    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:34.091087 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:34.091052    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:34.826615 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:34.826578    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:35.556310 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:35.556274    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:35.691453 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:35.691418    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:35.795613 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:35.795582    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:35.967986 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:35.967893    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:35.982809 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:35.982766    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24873d-flxrt"]
Sep 14 10:04:35.985622 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:35.985591    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24873d-flxrt"]
Sep 14 10:04:36.060607 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:36.060571    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:36.333628 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:36.333594    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:36.437643 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:36.437599    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:37.476348 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:37.476314    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:37.482429 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:37.482395    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a42d38e4-1fe2-4156-bc00-92bc19bcedbd" path="/var/lib/kubelet/pods/a42d38e4-1fe2-4156-bc00-92bc19bcedbd/volumes"
Sep 14 10:04:38.690552 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:38.690511    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:39.499243 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:39.499208    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:40.305761 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:40.305716    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:41.216200 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:41.216168    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:42.730277 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:42.730241    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:43.587799 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:43.587764    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:43.856915 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:43.856798    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:43.957434 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:43.957398    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:44.217448 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:44.217347    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-24873d-openshift-default-849786766d-xx9wt"]
Sep 14 10:04:44.217632 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:44.217590    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-24873d-openshift-default-849786766d-xx9wt" podUID="e67fbb48-e200-495b-8af4-590c28adf33d" containerName="istio-proxy" containerID="cri-o://34db4e6dcb113290aea30e7b7ce50568a5f9b2f600612d4dd5cdab3e0b5ff879" gracePeriod=30
Sep 14 10:04:44.254274 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:44.254227    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:44.502612 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:44.502572    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:44.689325 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:44.689286    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:44.965603 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:44.965498    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp"]
Sep 14 10:04:44.966150 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:44.966129    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a68cc340-1e5b-4971-96cc-91a62af7d642" containerName="istio-proxy"
Sep 14 10:04:44.966249 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:44.966153    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a68cc340-1e5b-4971-96cc-91a62af7d642" containerName="istio-proxy"
Sep 14 10:04:44.966249 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:44.966168    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d55a291-49c9-4c1e-8ae1-7cf6632d4395" containerName="main"
Sep 14 10:04:44.966249 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:44.966176    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d55a291-49c9-4c1e-8ae1-7cf6632d4395" containerName="main"
Sep 14 10:04:44.966249 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:44.966184    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53af90b8-85ec-4fa4-9831-59e5e6ce0037" containerName="main"
Sep 14 10:04:44.966249 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:44.966192    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="53af90b8-85ec-4fa4-9831-59e5e6ce0037" containerName="main"
Sep 14 10:04:44.966249 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:44.966204    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a42d38e4-1fe2-4156-bc00-92bc19bcedbd" containerName="revoke-keys"
Sep 14 10:04:44.966249 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:44.966213    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a42d38e4-1fe2-4156-bc00-92bc19bcedbd" containerName="revoke-keys"
Sep 14 10:04:44.966586 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:44.966319    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a68cc340-1e5b-4971-96cc-91a62af7d642" containerName="istio-proxy"
Sep 14 10:04:44.966586 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:44.966332    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a42d38e4-1fe2-4156-bc00-92bc19bcedbd" containerName="revoke-keys"
Sep 14 10:04:44.966586 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:44.966345    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="53af90b8-85ec-4fa4-9831-59e5e6ce0037" containerName="main"
Sep 14 10:04:44.966586 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:44.966353    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a68cc340-1e5b-4971-96cc-91a62af7d642" containerName="istio-proxy"
Sep 14 10:04:44.966586 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:44.966367    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d55a291-49c9-4c1e-8ae1-7cf6632d4395" containerName="main"
Sep 14 10:04:44.970485 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:44.970459    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp"
Sep 14 10:04:44.972805 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:44.972775    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-51aefd-openshift-default-dockercfg-tmsr9\""
Sep 14 10:04:44.983498 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:44.983463    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp"]
Sep 14 10:04:45.007303 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:45.007270    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:45.021714 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:45.021677    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1bb4690-068b-43c3-abdb-8bcda1591ae4-workload-socket\") pod \"e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp\" (UID: \"e1bb4690-068b-43c3-abdb-8bcda1591ae4\") " pod="openshift-ingress/e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp"
Sep 14 10:04:45.021868 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:45.021721    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1bb4690-068b-43c3-abdb-8bcda1591ae4-istio-podinfo\") pod \"e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp\" (UID: \"e1bb4690-068b-43c3-abdb-8bcda1591ae4\") " pod="openshift-ingress/e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp"
Sep 14 10:04:45.021868 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:45.021750    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1bb4690-068b-43c3-abdb-8bcda1591ae4-workload-certs\") pod \"e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp\" (UID: \"e1bb4690-068b-43c3-abdb-8bcda1591ae4\") " pod="openshift-ingress/e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp"
Sep 14 10:04:45.021868 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:45.021829    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1bb4690-068b-43c3-abdb-8bcda1591ae4-istio-data\") pod \"e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp\" (UID: \"e1bb4690-068b-43c3-abdb-8bcda1591ae4\") " pod="openshift-ingress/e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp"
Sep 14 10:04:45.022065 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:45.021934    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1bb4690-068b-43c3-abdb-8bcda1591ae4-istiod-ca-cert\") pod \"e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp\" (UID: \"e1bb4690-068b-43c3-abdb-8bcda1591ae4\") " pod="openshift-ingress/e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp"
Sep 14 10:04:45.022065 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:45.021992    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvgn8\" (UniqueName: \"kubernetes.io/projected/e1bb4690-068b-43c3-abdb-8bcda1591ae4-kube-api-access-cvgn8\") pod \"e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp\" (UID: \"e1bb4690-068b-43c3-abdb-8bcda1591ae4\") " pod="openshift-ingress/e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp"
Sep 14 10:04:45.022178 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:45.022078    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1bb4690-068b-43c3-abdb-8bcda1591ae4-istio-envoy\") pod \"e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp\" (UID: \"e1bb4690-068b-43c3-abdb-8bcda1591ae4\") " pod="openshift-ingress/e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp"
Sep 14 10:04:45.022178 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:45.022113    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1bb4690-068b-43c3-abdb-8bcda1591ae4-istio-token\") pod \"e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp\" (UID: \"e1bb4690-068b-43c3-abdb-8bcda1591ae4\") " pod="openshift-ingress/e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp"
Sep 14 10:04:45.022178 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:45.022151    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1bb4690-068b-43c3-abdb-8bcda1591ae4-credential-socket\") pod \"e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp\" (UID: \"e1bb4690-068b-43c3-abdb-8bcda1591ae4\") " pod="openshift-ingress/e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp"
Sep 14 10:04:45.122688 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:45.122650    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvgn8\" (UniqueName: \"kubernetes.io/projected/e1bb4690-068b-43c3-abdb-8bcda1591ae4-kube-api-access-cvgn8\") pod \"e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp\" (UID: \"e1bb4690-068b-43c3-abdb-8bcda1591ae4\") " pod="openshift-ingress/e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp"
Sep 14 10:04:45.122688 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:45.122688    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1bb4690-068b-43c3-abdb-8bcda1591ae4-istio-envoy\") pod \"e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp\" (UID: \"e1bb4690-068b-43c3-abdb-8bcda1591ae4\") " pod="openshift-ingress/e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp"
Sep 14 10:04:45.122966 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:45.122707    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1bb4690-068b-43c3-abdb-8bcda1591ae4-istio-token\") pod \"e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp\" (UID: \"e1bb4690-068b-43c3-abdb-8bcda1591ae4\") " pod="openshift-ingress/e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp"
Sep 14 10:04:45.122966 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:45.122835    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1bb4690-068b-43c3-abdb-8bcda1591ae4-credential-socket\") pod \"e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp\" (UID: \"e1bb4690-068b-43c3-abdb-8bcda1591ae4\") " pod="openshift-ingress/e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp"
Sep 14 10:04:45.123102 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:45.122992    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1bb4690-068b-43c3-abdb-8bcda1591ae4-workload-socket\") pod \"e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp\" (UID: \"e1bb4690-068b-43c3-abdb-8bcda1591ae4\") " pod="openshift-ingress/e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp"
Sep 14 10:04:45.123102 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:45.123044    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1bb4690-068b-43c3-abdb-8bcda1591ae4-istio-podinfo\") pod \"e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp\" (UID: \"e1bb4690-068b-43c3-abdb-8bcda1591ae4\") " pod="openshift-ingress/e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp"
Sep 14 10:04:45.123102 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:45.123085    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1bb4690-068b-43c3-abdb-8bcda1591ae4-workload-certs\") pod \"e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp\" (UID: \"e1bb4690-068b-43c3-abdb-8bcda1591ae4\") " pod="openshift-ingress/e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp"
Sep 14 10:04:45.123257 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:45.123137    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1bb4690-068b-43c3-abdb-8bcda1591ae4-istio-data\") pod \"e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp\" (UID: \"e1bb4690-068b-43c3-abdb-8bcda1591ae4\") " pod="openshift-ingress/e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp"
Sep 14 10:04:45.123257 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:45.123199    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1bb4690-068b-43c3-abdb-8bcda1591ae4-istiod-ca-cert\") pod \"e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp\" (UID: \"e1bb4690-068b-43c3-abdb-8bcda1591ae4\") " pod="openshift-ingress/e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp"
Sep 14 10:04:45.123359 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:45.123298    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1bb4690-068b-43c3-abdb-8bcda1591ae4-credential-socket\") pod \"e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp\" (UID: \"e1bb4690-068b-43c3-abdb-8bcda1591ae4\") " pod="openshift-ingress/e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp"
Sep 14 10:04:45.123516 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:45.123491    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1bb4690-068b-43c3-abdb-8bcda1591ae4-workload-socket\") pod \"e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp\" (UID: \"e1bb4690-068b-43c3-abdb-8bcda1591ae4\") " pod="openshift-ingress/e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp"
Sep 14 10:04:45.123595 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:45.123517    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1bb4690-068b-43c3-abdb-8bcda1591ae4-workload-certs\") pod \"e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp\" (UID: \"e1bb4690-068b-43c3-abdb-8bcda1591ae4\") " pod="openshift-ingress/e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp"
Sep 14 10:04:45.123595 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:45.123583    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1bb4690-068b-43c3-abdb-8bcda1591ae4-istio-data\") pod \"e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp\" (UID: \"e1bb4690-068b-43c3-abdb-8bcda1591ae4\") " pod="openshift-ingress/e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp"
Sep 14 10:04:45.123911 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:45.123890    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1bb4690-068b-43c3-abdb-8bcda1591ae4-istiod-ca-cert\") pod \"e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp\" (UID: \"e1bb4690-068b-43c3-abdb-8bcda1591ae4\") " pod="openshift-ingress/e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp"
Sep 14 10:04:45.125173 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:45.125141    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1bb4690-068b-43c3-abdb-8bcda1591ae4-istio-envoy\") pod \"e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp\" (UID: \"e1bb4690-068b-43c3-abdb-8bcda1591ae4\") " pod="openshift-ingress/e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp"
Sep 14 10:04:45.125505 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:45.125485    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1bb4690-068b-43c3-abdb-8bcda1591ae4-istio-podinfo\") pod \"e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp\" (UID: \"e1bb4690-068b-43c3-abdb-8bcda1591ae4\") " pod="openshift-ingress/e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp"
Sep 14 10:04:45.134328 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:45.134290    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1bb4690-068b-43c3-abdb-8bcda1591ae4-istio-token\") pod \"e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp\" (UID: \"e1bb4690-068b-43c3-abdb-8bcda1591ae4\") " pod="openshift-ingress/e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp"
Sep 14 10:04:45.136985 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:45.136953    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvgn8\" (UniqueName: \"kubernetes.io/projected/e1bb4690-068b-43c3-abdb-8bcda1591ae4-kube-api-access-cvgn8\") pod \"e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp\" (UID: \"e1bb4690-068b-43c3-abdb-8bcda1591ae4\") " pod="openshift-ingress/e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp"
Sep 14 10:04:45.284352 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:45.284312    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp"
Sep 14 10:04:45.345442 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:45.343981    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:45.518793 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:45.518739    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:45.572866 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:45.572838    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp"]
Sep 14 10:04:45.575219 ip-10-0-132-4 kubenswrapper[2563]: W0914 10:04:45.575186    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1bb4690_068b_43c3_abdb_8bcda1591ae4.slice/crio-81428c8f44c6225ed5125de90e4955519a373816313d58488b943214f74fafb2 WatchSource:0}: Error finding container 81428c8f44c6225ed5125de90e4955519a373816313d58488b943214f74fafb2: Status 404 returned error can't find the container with id 81428c8f44c6225ed5125de90e4955519a373816313d58488b943214f74fafb2
Sep 14 10:04:45.578211 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:45.578170    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 10:04:45.578344 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:45.578265    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 10:04:45.578344 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:45.578313    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 10:04:46.236829 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:46.236761    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp" event={"ID":"e1bb4690-068b-43c3-abdb-8bcda1591ae4","Type":"ContainerStarted","Data":"2f244eaaaa673331a39611fe59a0f1a394dab63f034769c86e833c709e2d4aaa"}
Sep 14 10:04:46.237362 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:46.236863    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp" event={"ID":"e1bb4690-068b-43c3-abdb-8bcda1591ae4","Type":"ContainerStarted","Data":"81428c8f44c6225ed5125de90e4955519a373816313d58488b943214f74fafb2"}
Sep 14 10:04:46.269336 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:46.269272    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp" podStartSLOduration=2.269248773 podStartE2EDuration="2.269248773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:04:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:04:46.268520946 +0000 UTC m=+1275.308195566" watchObservedRunningTime="2026-09-14 10:04:46.269248773 +0000 UTC m=+1275.308923397"
Sep 14 10:04:46.285239 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:46.285198    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp"
Sep 14 10:04:46.291198 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:46.291163    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp"
Sep 14 10:04:46.341390 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:46.341338    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:46.867546 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:46.867504    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:47.136079 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:47.135958    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:47.244576 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:47.244528    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp"
Sep 14 10:04:47.245828 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:47.245804    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp"
Sep 14 10:04:47.569602 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:47.569559    2563 patch_prober.go:28] interesting pod/e2e-inf-b-24873d-openshift-default-849786766d-xx9wt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 14 10:04:47.569782 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:47.569634    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-24873d-openshift-default-849786766d-xx9wt" podUID="e67fbb48-e200-495b-8af4-590c28adf33d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 14 10:04:48.020073 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:48.020019    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:48.253300 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:48.253264    2563 generic.go:356] "Generic (PLEG): container finished" podID="f7b2e299-2b37-4d21-9264-1cbf9d15c274" containerID="bd56ee38bd9e54dada49c06a4a0c2cc9a307672c3314a785c10e1a1ad74e855b" exitCode=0
Sep 14 10:04:48.253730 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:48.253336    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5" event={"ID":"f7b2e299-2b37-4d21-9264-1cbf9d15c274","Type":"ContainerDied","Data":"bd56ee38bd9e54dada49c06a4a0c2cc9a307672c3314a785c10e1a1ad74e855b"}
Sep 14 10:04:48.254824 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:48.254498    2563 scope.go:117] "RemoveContainer" containerID="bd56ee38bd9e54dada49c06a4a0c2cc9a307672c3314a785c10e1a1ad74e855b"
Sep 14 10:04:48.262076 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:48.262036    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 10:04:48.262212 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:48.262117    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 10:04:48.262212 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:48.262148    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 10:04:48.348480 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:48.348449    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:48.461310 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:48.461275    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:49.014847 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:49.014810    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:49.262931 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:49.262889    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5" event={"ID":"f7b2e299-2b37-4d21-9264-1cbf9d15c274","Type":"ContainerStarted","Data":"f93d12956a27cd5d8b73982ee0e944d15d2f7c9835d9e4fc25187c339cf5be1c"}
Sep 14 10:04:49.558102 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:49.558070    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-24873d-openshift-default-849786766d-xx9wt"
Sep 14 10:04:49.677832 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:49.677731    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e67fbb48-e200-495b-8af4-590c28adf33d-istio-envoy\") pod \"e67fbb48-e200-495b-8af4-590c28adf33d\" (UID: \"e67fbb48-e200-495b-8af4-590c28adf33d\") "
Sep 14 10:04:49.677832 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:49.677789    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e67fbb48-e200-495b-8af4-590c28adf33d-istio-data\") pod \"e67fbb48-e200-495b-8af4-590c28adf33d\" (UID: \"e67fbb48-e200-495b-8af4-590c28adf33d\") "
Sep 14 10:04:49.677832 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:49.677809    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e67fbb48-e200-495b-8af4-590c28adf33d-istiod-ca-cert\") pod \"e67fbb48-e200-495b-8af4-590c28adf33d\" (UID: \"e67fbb48-e200-495b-8af4-590c28adf33d\") "
Sep 14 10:04:49.677832 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:49.677831    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e67fbb48-e200-495b-8af4-590c28adf33d-workload-certs\") pod \"e67fbb48-e200-495b-8af4-590c28adf33d\" (UID: \"e67fbb48-e200-495b-8af4-590c28adf33d\") "
Sep 14 10:04:49.678222 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:49.677882    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e67fbb48-e200-495b-8af4-590c28adf33d-istio-token\") pod \"e67fbb48-e200-495b-8af4-590c28adf33d\" (UID: \"e67fbb48-e200-495b-8af4-590c28adf33d\") "
Sep 14 10:04:49.678222 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:49.677905    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e67fbb48-e200-495b-8af4-590c28adf33d-credential-socket\") pod \"e67fbb48-e200-495b-8af4-590c28adf33d\" (UID: \"e67fbb48-e200-495b-8af4-590c28adf33d\") "
Sep 14 10:04:49.678222 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:49.677941    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e67fbb48-e200-495b-8af4-590c28adf33d-istio-podinfo\") pod \"e67fbb48-e200-495b-8af4-590c28adf33d\" (UID: \"e67fbb48-e200-495b-8af4-590c28adf33d\") "
Sep 14 10:04:49.678222 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:49.677986    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h28nx\" (UniqueName: \"kubernetes.io/projected/e67fbb48-e200-495b-8af4-590c28adf33d-kube-api-access-h28nx\") pod \"e67fbb48-e200-495b-8af4-590c28adf33d\" (UID: \"e67fbb48-e200-495b-8af4-590c28adf33d\") "
Sep 14 10:04:49.678222 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:49.678075    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e67fbb48-e200-495b-8af4-590c28adf33d-workload-socket\") pod \"e67fbb48-e200-495b-8af4-590c28adf33d\" (UID: \"e67fbb48-e200-495b-8af4-590c28adf33d\") "
Sep 14 10:04:49.678222 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:49.678104    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e67fbb48-e200-495b-8af4-590c28adf33d-istio-data" (OuterVolumeSpecName: "istio-data") pod "e67fbb48-e200-495b-8af4-590c28adf33d" (UID: "e67fbb48-e200-495b-8af4-590c28adf33d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:04:49.678222 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:49.678180    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e67fbb48-e200-495b-8af4-590c28adf33d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e67fbb48-e200-495b-8af4-590c28adf33d" (UID: "e67fbb48-e200-495b-8af4-590c28adf33d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:04:49.678222 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:49.678202    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e67fbb48-e200-495b-8af4-590c28adf33d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e67fbb48-e200-495b-8af4-590c28adf33d" (UID: "e67fbb48-e200-495b-8af4-590c28adf33d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:04:49.678647 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:49.678390    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e67fbb48-e200-495b-8af4-590c28adf33d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e67fbb48-e200-495b-8af4-590c28adf33d" (UID: "e67fbb48-e200-495b-8af4-590c28adf33d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:04:49.678647 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:49.678405    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e67fbb48-e200-495b-8af4-590c28adf33d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e67fbb48-e200-495b-8af4-590c28adf33d" (UID: "e67fbb48-e200-495b-8af4-590c28adf33d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:04:49.678647 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:49.678524    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e67fbb48-e200-495b-8af4-590c28adf33d-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:04:49.678647 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:49.678540    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e67fbb48-e200-495b-8af4-590c28adf33d-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:04:49.678647 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:49.678551    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e67fbb48-e200-495b-8af4-590c28adf33d-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:04:49.678647 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:49.678562    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e67fbb48-e200-495b-8af4-590c28adf33d-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:04:49.678647 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:49.678573    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e67fbb48-e200-495b-8af4-590c28adf33d-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:04:49.680667 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:49.680387    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e67fbb48-e200-495b-8af4-590c28adf33d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e67fbb48-e200-495b-8af4-590c28adf33d" (UID: "e67fbb48-e200-495b-8af4-590c28adf33d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 10:04:49.680667 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:49.680415    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e67fbb48-e200-495b-8af4-590c28adf33d-kube-api-access-h28nx" (OuterVolumeSpecName: "kube-api-access-h28nx") pod "e67fbb48-e200-495b-8af4-590c28adf33d" (UID: "e67fbb48-e200-495b-8af4-590c28adf33d"). InnerVolumeSpecName "kube-api-access-h28nx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:04:49.680873 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:49.680717    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e67fbb48-e200-495b-8af4-590c28adf33d-istio-token" (OuterVolumeSpecName: "istio-token") pod "e67fbb48-e200-495b-8af4-590c28adf33d" (UID: "e67fbb48-e200-495b-8af4-590c28adf33d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:04:49.680873 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:49.680716    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e67fbb48-e200-495b-8af4-590c28adf33d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e67fbb48-e200-495b-8af4-590c28adf33d" (UID: "e67fbb48-e200-495b-8af4-590c28adf33d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:04:49.780154 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:49.780102    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e67fbb48-e200-495b-8af4-590c28adf33d-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:04:49.780154 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:49.780142    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e67fbb48-e200-495b-8af4-590c28adf33d-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:04:49.780154 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:49.780154    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e67fbb48-e200-495b-8af4-590c28adf33d-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:04:49.780154 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:49.780166    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h28nx\" (UniqueName: \"kubernetes.io/projected/e67fbb48-e200-495b-8af4-590c28adf33d-kube-api-access-h28nx\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:04:49.871113 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:49.871072    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5"
Sep 14 10:04:49.871461 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:49.871435    2563 patch_prober.go:28] interesting pod/e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Sep 14 10:04:49.871556 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:49.871496    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5" podUID="f7b2e299-2b37-4d21-9264-1cbf9d15c274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Sep 14 10:04:49.908153 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:49.908114    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:50.271624 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:50.271585    2563 generic.go:356] "Generic (PLEG): container finished" podID="e67fbb48-e200-495b-8af4-590c28adf33d" containerID="34db4e6dcb113290aea30e7b7ce50568a5f9b2f600612d4dd5cdab3e0b5ff879" exitCode=0
Sep 14 10:04:50.272133 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:50.271640    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-24873d-openshift-default-849786766d-xx9wt"
Sep 14 10:04:50.272133 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:50.271664    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-24873d-openshift-default-849786766d-xx9wt" event={"ID":"e67fbb48-e200-495b-8af4-590c28adf33d","Type":"ContainerDied","Data":"34db4e6dcb113290aea30e7b7ce50568a5f9b2f600612d4dd5cdab3e0b5ff879"}
Sep 14 10:04:50.272133 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:50.271736    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-24873d-openshift-default-849786766d-xx9wt" event={"ID":"e67fbb48-e200-495b-8af4-590c28adf33d","Type":"ContainerDied","Data":"ba5199078dc0609ff936dc1b7adbd3a907d69b1586dba1d6648b1854af751704"}
Sep 14 10:04:50.272133 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:50.271788    2563 scope.go:117] "RemoveContainer" containerID="34db4e6dcb113290aea30e7b7ce50568a5f9b2f600612d4dd5cdab3e0b5ff879"
Sep 14 10:04:50.285417 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:50.285387    2563 scope.go:117] "RemoveContainer" containerID="115cae8af62489681ec015d8bd639be57efee89686e43d7f9ee379018d7c82a3"
Sep 14 10:04:50.295678 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:50.295648    2563 scope.go:117] "RemoveContainer" containerID="34db4e6dcb113290aea30e7b7ce50568a5f9b2f600612d4dd5cdab3e0b5ff879"
Sep 14 10:04:50.296111 ip-10-0-132-4 kubenswrapper[2563]: E0914 10:04:50.296086    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34db4e6dcb113290aea30e7b7ce50568a5f9b2f600612d4dd5cdab3e0b5ff879\": container with ID starting with 34db4e6dcb113290aea30e7b7ce50568a5f9b2f600612d4dd5cdab3e0b5ff879 not found: ID does not exist" containerID="34db4e6dcb113290aea30e7b7ce50568a5f9b2f600612d4dd5cdab3e0b5ff879"
Sep 14 10:04:50.296201 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:50.296120    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34db4e6dcb113290aea30e7b7ce50568a5f9b2f600612d4dd5cdab3e0b5ff879"} err="failed to get container status \"34db4e6dcb113290aea30e7b7ce50568a5f9b2f600612d4dd5cdab3e0b5ff879\": rpc error: code = NotFound desc = could not find container \"34db4e6dcb113290aea30e7b7ce50568a5f9b2f600612d4dd5cdab3e0b5ff879\": container with ID starting with 34db4e6dcb113290aea30e7b7ce50568a5f9b2f600612d4dd5cdab3e0b5ff879 not found: ID does not exist"
Sep 14 10:04:50.296201 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:50.296139    2563 scope.go:117] "RemoveContainer" containerID="115cae8af62489681ec015d8bd639be57efee89686e43d7f9ee379018d7c82a3"
Sep 14 10:04:50.296479 ip-10-0-132-4 kubenswrapper[2563]: E0914 10:04:50.296453    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"115cae8af62489681ec015d8bd639be57efee89686e43d7f9ee379018d7c82a3\": container with ID starting with 115cae8af62489681ec015d8bd639be57efee89686e43d7f9ee379018d7c82a3 not found: ID does not exist" containerID="115cae8af62489681ec015d8bd639be57efee89686e43d7f9ee379018d7c82a3"
Sep 14 10:04:50.296520 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:50.296492    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"115cae8af62489681ec015d8bd639be57efee89686e43d7f9ee379018d7c82a3"} err="failed to get container status \"115cae8af62489681ec015d8bd639be57efee89686e43d7f9ee379018d7c82a3\": rpc error: code = NotFound desc = could not find container \"115cae8af62489681ec015d8bd639be57efee89686e43d7f9ee379018d7c82a3\": container with ID starting with 115cae8af62489681ec015d8bd639be57efee89686e43d7f9ee379018d7c82a3 not found: ID does not exist"
Sep 14 10:04:50.313872 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:50.313822    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-24873d-openshift-default-849786766d-xx9wt"]
Sep 14 10:04:50.342028 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:50.341290    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-24873d-openshift-default-849786766d-xx9wt"]
Sep 14 10:04:50.871600 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:50.871562    2563 patch_prober.go:28] interesting pod/e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Sep 14 10:04:50.871789 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:50.871623    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5" podUID="f7b2e299-2b37-4d21-9264-1cbf9d15c274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Sep 14 10:04:51.069259 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:51.069219    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:51.483676 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:51.483633    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e67fbb48-e200-495b-8af4-590c28adf33d" path="/var/lib/kubelet/pods/e67fbb48-e200-495b-8af4-590c28adf33d/volumes"
Sep 14 10:04:51.871666 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:51.871621    2563 patch_prober.go:28] interesting pod/e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Sep 14 10:04:51.871885 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:51.871686    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5" podUID="f7b2e299-2b37-4d21-9264-1cbf9d15c274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Sep 14 10:04:52.032044 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:52.031941    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:52.351496 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:52.351451    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-b6812e/distinct-2-w0-b6812e-kserve-7ffbdbf7d-kx89t"]
Sep 14 10:04:52.351947 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:52.351924    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e67fbb48-e200-495b-8af4-590c28adf33d" containerName="istio-proxy"
Sep 14 10:04:52.352130 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:52.352055    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e67fbb48-e200-495b-8af4-590c28adf33d" containerName="istio-proxy"
Sep 14 10:04:52.352130 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:52.352093    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e67fbb48-e200-495b-8af4-590c28adf33d" containerName="istio-proxy"
Sep 14 10:04:52.352130 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:52.352102    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e67fbb48-e200-495b-8af4-590c28adf33d" containerName="istio-proxy"
Sep 14 10:04:52.352260 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:52.352137    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a68cc340-1e5b-4971-96cc-91a62af7d642" containerName="istio-proxy"
Sep 14 10:04:52.352260 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:52.352144    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a68cc340-1e5b-4971-96cc-91a62af7d642" containerName="istio-proxy"
Sep 14 10:04:52.352327 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:52.352262    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e67fbb48-e200-495b-8af4-590c28adf33d" containerName="istio-proxy"
Sep 14 10:04:52.352327 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:52.352272    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e67fbb48-e200-495b-8af4-590c28adf33d" containerName="istio-proxy"
Sep 14 10:04:52.355323 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:52.355299    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b6812e/distinct-2-w0-b6812e-kserve-7ffbdbf7d-kx89t"
Sep 14 10:04:52.357733 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:52.357701    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b6812e\"/\"distinct-2-w0-b6812e-kserve-self-signed-certs\""
Sep 14 10:04:52.370508 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:52.370469    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b6812e/distinct-2-w0-b6812e-kserve-7ffbdbf7d-kx89t"]
Sep 14 10:04:52.409243 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:52.409199    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2-home\") pod \"distinct-2-w0-b6812e-kserve-7ffbdbf7d-kx89t\" (UID: \"dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2\") " pod="e2e-models-e2e-worker-w0-b6812e/distinct-2-w0-b6812e-kserve-7ffbdbf7d-kx89t"
Sep 14 10:04:52.409429 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:52.409258    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2-tmp-dir\") pod \"distinct-2-w0-b6812e-kserve-7ffbdbf7d-kx89t\" (UID: \"dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2\") " pod="e2e-models-e2e-worker-w0-b6812e/distinct-2-w0-b6812e-kserve-7ffbdbf7d-kx89t"
Sep 14 10:04:52.409429 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:52.409364    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2-tls-certs\") pod \"distinct-2-w0-b6812e-kserve-7ffbdbf7d-kx89t\" (UID: \"dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2\") " pod="e2e-models-e2e-worker-w0-b6812e/distinct-2-w0-b6812e-kserve-7ffbdbf7d-kx89t"
Sep 14 10:04:52.409429 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:52.409396    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ft8c\" (UniqueName: \"kubernetes.io/projected/dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2-kube-api-access-4ft8c\") pod \"distinct-2-w0-b6812e-kserve-7ffbdbf7d-kx89t\" (UID: \"dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2\") " pod="e2e-models-e2e-worker-w0-b6812e/distinct-2-w0-b6812e-kserve-7ffbdbf7d-kx89t"
Sep 14 10:04:52.409563 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:52.409476    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2-dshm\") pod \"distinct-2-w0-b6812e-kserve-7ffbdbf7d-kx89t\" (UID: \"dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2\") " pod="e2e-models-e2e-worker-w0-b6812e/distinct-2-w0-b6812e-kserve-7ffbdbf7d-kx89t"
Sep 14 10:04:52.409563 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:52.409543    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2-model-cache\") pod \"distinct-2-w0-b6812e-kserve-7ffbdbf7d-kx89t\" (UID: \"dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2\") " pod="e2e-models-e2e-worker-w0-b6812e/distinct-2-w0-b6812e-kserve-7ffbdbf7d-kx89t"
Sep 14 10:04:52.475310 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:52.475278    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:52.510893 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:52.510852    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2-dshm\") pod \"distinct-2-w0-b6812e-kserve-7ffbdbf7d-kx89t\" (UID: \"dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2\") " pod="e2e-models-e2e-worker-w0-b6812e/distinct-2-w0-b6812e-kserve-7ffbdbf7d-kx89t"
Sep 14 10:04:52.511334 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:52.510902    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2-model-cache\") pod \"distinct-2-w0-b6812e-kserve-7ffbdbf7d-kx89t\" (UID: \"dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2\") " pod="e2e-models-e2e-worker-w0-b6812e/distinct-2-w0-b6812e-kserve-7ffbdbf7d-kx89t"
Sep 14 10:04:52.511334 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:52.510939    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2-home\") pod \"distinct-2-w0-b6812e-kserve-7ffbdbf7d-kx89t\" (UID: \"dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2\") " pod="e2e-models-e2e-worker-w0-b6812e/distinct-2-w0-b6812e-kserve-7ffbdbf7d-kx89t"
Sep 14 10:04:52.511334 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:52.510973    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2-tmp-dir\") pod \"distinct-2-w0-b6812e-kserve-7ffbdbf7d-kx89t\" (UID: \"dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2\") " pod="e2e-models-e2e-worker-w0-b6812e/distinct-2-w0-b6812e-kserve-7ffbdbf7d-kx89t"
Sep 14 10:04:52.511334 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:52.511127    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2-tls-certs\") pod \"distinct-2-w0-b6812e-kserve-7ffbdbf7d-kx89t\" (UID: \"dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2\") " pod="e2e-models-e2e-worker-w0-b6812e/distinct-2-w0-b6812e-kserve-7ffbdbf7d-kx89t"
Sep 14 10:04:52.511334 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:52.511171    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ft8c\" (UniqueName: \"kubernetes.io/projected/dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2-kube-api-access-4ft8c\") pod \"distinct-2-w0-b6812e-kserve-7ffbdbf7d-kx89t\" (UID: \"dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2\") " pod="e2e-models-e2e-worker-w0-b6812e/distinct-2-w0-b6812e-kserve-7ffbdbf7d-kx89t"
Sep 14 10:04:52.511598 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:52.511436    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2-model-cache\") pod \"distinct-2-w0-b6812e-kserve-7ffbdbf7d-kx89t\" (UID: \"dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2\") " pod="e2e-models-e2e-worker-w0-b6812e/distinct-2-w0-b6812e-kserve-7ffbdbf7d-kx89t"
Sep 14 10:04:52.511598 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:52.511471    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2-home\") pod \"distinct-2-w0-b6812e-kserve-7ffbdbf7d-kx89t\" (UID: \"dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2\") " pod="e2e-models-e2e-worker-w0-b6812e/distinct-2-w0-b6812e-kserve-7ffbdbf7d-kx89t"
Sep 14 10:04:52.511598 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:52.511496    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2-tmp-dir\") pod \"distinct-2-w0-b6812e-kserve-7ffbdbf7d-kx89t\" (UID: \"dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2\") " pod="e2e-models-e2e-worker-w0-b6812e/distinct-2-w0-b6812e-kserve-7ffbdbf7d-kx89t"
Sep 14 10:04:52.513522 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:52.513488    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2-dshm\") pod \"distinct-2-w0-b6812e-kserve-7ffbdbf7d-kx89t\" (UID: \"dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2\") " pod="e2e-models-e2e-worker-w0-b6812e/distinct-2-w0-b6812e-kserve-7ffbdbf7d-kx89t"
Sep 14 10:04:52.513662 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:52.513610    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2-tls-certs\") pod \"distinct-2-w0-b6812e-kserve-7ffbdbf7d-kx89t\" (UID: \"dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2\") " pod="e2e-models-e2e-worker-w0-b6812e/distinct-2-w0-b6812e-kserve-7ffbdbf7d-kx89t"
Sep 14 10:04:52.534412 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:52.531955    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ft8c\" (UniqueName: \"kubernetes.io/projected/dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2-kube-api-access-4ft8c\") pod \"distinct-2-w0-b6812e-kserve-7ffbdbf7d-kx89t\" (UID: \"dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2\") " pod="e2e-models-e2e-worker-w0-b6812e/distinct-2-w0-b6812e-kserve-7ffbdbf7d-kx89t"
Sep 14 10:04:52.665124 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:52.665024    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b6812e/distinct-2-w0-b6812e-kserve-7ffbdbf7d-kx89t"
Sep 14 10:04:52.871507 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:52.871461    2563 patch_prober.go:28] interesting pod/e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Sep 14 10:04:52.871718 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:52.871529    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5" podUID="f7b2e299-2b37-4d21-9264-1cbf9d15c274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Sep 14 10:04:52.977996 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:52.977960    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b6812e/distinct-2-w0-b6812e-kserve-7ffbdbf7d-kx89t"]
Sep 14 10:04:52.978719 ip-10-0-132-4 kubenswrapper[2563]: W0914 10:04:52.978693    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddedd7da3_0dff_4eb7_bc44_4aac8d9aa5b2.slice/crio-05e2ec598c6e7be0bc1a45ffe3baacf6435ec567c0d6ebd32423888f1d6c99e8 WatchSource:0}: Error finding container 05e2ec598c6e7be0bc1a45ffe3baacf6435ec567c0d6ebd32423888f1d6c99e8: Status 404 returned error can't find the container with id 05e2ec598c6e7be0bc1a45ffe3baacf6435ec567c0d6ebd32423888f1d6c99e8
Sep 14 10:04:53.108942 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:53.108908    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:53.295928 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:53.295890    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b6812e/distinct-2-w0-b6812e-kserve-7ffbdbf7d-kx89t" event={"ID":"dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2","Type":"ContainerStarted","Data":"8fbd676b89d67b475bd7b4b13026f6a6bba136393d4bcc60b6997f6c957aea73"}
Sep 14 10:04:53.295928 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:53.295935    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b6812e/distinct-2-w0-b6812e-kserve-7ffbdbf7d-kx89t" event={"ID":"dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2","Type":"ContainerStarted","Data":"05e2ec598c6e7be0bc1a45ffe3baacf6435ec567c0d6ebd32423888f1d6c99e8"}
Sep 14 10:04:53.316631 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:53.316572    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-b6812e/distinct-2-w0-b6812e-kserve-7ffbdbf7d-kx89t" podStartSLOduration=1.316548841 podStartE2EDuration="1.316548841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:04:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:04:53.312868615 +0000 UTC m=+1282.352543235" watchObservedRunningTime="2026-09-14 10:04:53.316548841 +0000 UTC m=+1282.356223462"
Sep 14 10:04:53.871865 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:53.871829    2563 patch_prober.go:28] interesting pod/e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Sep 14 10:04:53.872313 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:53.872108    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5" podUID="f7b2e299-2b37-4d21-9264-1cbf9d15c274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Sep 14 10:04:54.016711 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:54.016620    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:54.086613 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:54.086536    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:54.616275 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:54.616236    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:54.740478 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:54.740424    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:54.841204 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:54.841171    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:54.877250 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:54.877161    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5"
Sep 14 10:04:54.877729 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:54.877714    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5"
Sep 14 10:04:54.878595 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:54.878571    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5"
Sep 14 10:04:56.698402 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:56.698362    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:56.822407 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:56.822371    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:04:57.236560 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:04:57.236502    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:05:02.665528 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:02.665481    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-b6812e/distinct-2-w0-b6812e-kserve-7ffbdbf7d-kx89t"
Sep 14 10:05:02.666064 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:02.665552    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b6812e/distinct-2-w0-b6812e-kserve-7ffbdbf7d-kx89t"
Sep 14 10:05:02.679113 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:02.679078    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-b6812e/distinct-2-w0-b6812e-kserve-7ffbdbf7d-kx89t"
Sep 14 10:05:03.375050 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:03.374996    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b6812e/distinct-2-w0-b6812e-kserve-7ffbdbf7d-kx89t"
Sep 14 10:05:16.503897 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:16.503853    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:05:17.418344 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:17.418285    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:05:18.173501 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:18.173459    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:05:18.286210 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:18.286163    2563 patch_prober.go:28] interesting pod/e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 14 10:05:18.287718 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:18.286243    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp" podUID="e1bb4690-068b-43c3-abdb-8bcda1591ae4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 14 10:05:18.306397 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:18.306352    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:05:18.483763 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:18.483673    2563 generic.go:356] "Generic (PLEG): container finished" podID="e1bb4690-068b-43c3-abdb-8bcda1591ae4" containerID="2f244eaaaa673331a39611fe59a0f1a394dab63f034769c86e833c709e2d4aaa" exitCode=0
Sep 14 10:05:18.483922 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:18.483748    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp" event={"ID":"e1bb4690-068b-43c3-abdb-8bcda1591ae4","Type":"ContainerDied","Data":"2f244eaaaa673331a39611fe59a0f1a394dab63f034769c86e833c709e2d4aaa"}
Sep 14 10:05:18.484258 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:18.484242    2563 scope.go:117] "RemoveContainer" containerID="2f244eaaaa673331a39611fe59a0f1a394dab63f034769c86e833c709e2d4aaa"
Sep 14 10:05:18.485782 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:18.485720    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 10:05:18.485858 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:18.485820    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 10:05:18.485905 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:18.485863    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 10:05:18.678626 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:18.678589    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:05:18.955450 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:18.955415    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl"]
Sep 14 10:05:18.958351 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:18.958322    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl"
Sep 14 10:05:18.960367 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:18.960337    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-f4d054-openshift-default-dockercfg-cbtwm\""
Sep 14 10:05:18.972345 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:18.972301    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl"]
Sep 14 10:05:19.011863 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:19.011823    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:05:19.031122 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:19.031079    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d80c24f-867f-4c36-9e8b-3ab262ef1fb7-istio-token\") pod \"e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl\" (UID: \"3d80c24f-867f-4c36-9e8b-3ab262ef1fb7\") " pod="openshift-ingress/e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl"
Sep 14 10:05:19.031122 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:19.031124    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d80c24f-867f-4c36-9e8b-3ab262ef1fb7-istio-podinfo\") pod \"e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl\" (UID: \"3d80c24f-867f-4c36-9e8b-3ab262ef1fb7\") " pod="openshift-ingress/e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl"
Sep 14 10:05:19.031356 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:19.031221    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d80c24f-867f-4c36-9e8b-3ab262ef1fb7-istiod-ca-cert\") pod \"e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl\" (UID: \"3d80c24f-867f-4c36-9e8b-3ab262ef1fb7\") " pod="openshift-ingress/e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl"
Sep 14 10:05:19.031356 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:19.031243    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d80c24f-867f-4c36-9e8b-3ab262ef1fb7-istio-envoy\") pod \"e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl\" (UID: \"3d80c24f-867f-4c36-9e8b-3ab262ef1fb7\") " pod="openshift-ingress/e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl"
Sep 14 10:05:19.031356 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:19.031260    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d80c24f-867f-4c36-9e8b-3ab262ef1fb7-istio-data\") pod \"e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl\" (UID: \"3d80c24f-867f-4c36-9e8b-3ab262ef1fb7\") " pod="openshift-ingress/e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl"
Sep 14 10:05:19.031356 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:19.031286    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d80c24f-867f-4c36-9e8b-3ab262ef1fb7-workload-socket\") pod \"e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl\" (UID: \"3d80c24f-867f-4c36-9e8b-3ab262ef1fb7\") " pod="openshift-ingress/e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl"
Sep 14 10:05:19.031495 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:19.031434    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d80c24f-867f-4c36-9e8b-3ab262ef1fb7-credential-socket\") pod \"e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl\" (UID: \"3d80c24f-867f-4c36-9e8b-3ab262ef1fb7\") " pod="openshift-ingress/e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl"
Sep 14 10:05:19.031534 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:19.031512    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d80c24f-867f-4c36-9e8b-3ab262ef1fb7-workload-certs\") pod \"e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl\" (UID: \"3d80c24f-867f-4c36-9e8b-3ab262ef1fb7\") " pod="openshift-ingress/e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl"
Sep 14 10:05:19.031570 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:19.031547    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29szt\" (UniqueName: \"kubernetes.io/projected/3d80c24f-867f-4c36-9e8b-3ab262ef1fb7-kube-api-access-29szt\") pod \"e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl\" (UID: \"3d80c24f-867f-4c36-9e8b-3ab262ef1fb7\") " pod="openshift-ingress/e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl"
Sep 14 10:05:19.132471 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:19.132425    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d80c24f-867f-4c36-9e8b-3ab262ef1fb7-istiod-ca-cert\") pod \"e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl\" (UID: \"3d80c24f-867f-4c36-9e8b-3ab262ef1fb7\") " pod="openshift-ingress/e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl"
Sep 14 10:05:19.132471 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:19.132471    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d80c24f-867f-4c36-9e8b-3ab262ef1fb7-istio-envoy\") pod \"e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl\" (UID: \"3d80c24f-867f-4c36-9e8b-3ab262ef1fb7\") " pod="openshift-ingress/e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl"
Sep 14 10:05:19.132722 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:19.132487    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d80c24f-867f-4c36-9e8b-3ab262ef1fb7-istio-data\") pod \"e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl\" (UID: \"3d80c24f-867f-4c36-9e8b-3ab262ef1fb7\") " pod="openshift-ingress/e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl"
Sep 14 10:05:19.132722 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:19.132609    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d80c24f-867f-4c36-9e8b-3ab262ef1fb7-workload-socket\") pod \"e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl\" (UID: \"3d80c24f-867f-4c36-9e8b-3ab262ef1fb7\") " pod="openshift-ingress/e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl"
Sep 14 10:05:19.132722 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:19.132662    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d80c24f-867f-4c36-9e8b-3ab262ef1fb7-credential-socket\") pod \"e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl\" (UID: \"3d80c24f-867f-4c36-9e8b-3ab262ef1fb7\") " pod="openshift-ingress/e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl"
Sep 14 10:05:19.132885 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:19.132722    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d80c24f-867f-4c36-9e8b-3ab262ef1fb7-workload-certs\") pod \"e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl\" (UID: \"3d80c24f-867f-4c36-9e8b-3ab262ef1fb7\") " pod="openshift-ingress/e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl"
Sep 14 10:05:19.132885 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:19.132756    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29szt\" (UniqueName: \"kubernetes.io/projected/3d80c24f-867f-4c36-9e8b-3ab262ef1fb7-kube-api-access-29szt\") pod \"e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl\" (UID: \"3d80c24f-867f-4c36-9e8b-3ab262ef1fb7\") " pod="openshift-ingress/e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl"
Sep 14 10:05:19.132983 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:19.132896    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d80c24f-867f-4c36-9e8b-3ab262ef1fb7-istio-token\") pod \"e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl\" (UID: \"3d80c24f-867f-4c36-9e8b-3ab262ef1fb7\") " pod="openshift-ingress/e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl"
Sep 14 10:05:19.132983 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:19.132928    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d80c24f-867f-4c36-9e8b-3ab262ef1fb7-istio-podinfo\") pod \"e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl\" (UID: \"3d80c24f-867f-4c36-9e8b-3ab262ef1fb7\") " pod="openshift-ingress/e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl"
Sep 14 10:05:19.133119 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:19.132995    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d80c24f-867f-4c36-9e8b-3ab262ef1fb7-istio-data\") pod \"e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl\" (UID: \"3d80c24f-867f-4c36-9e8b-3ab262ef1fb7\") " pod="openshift-ingress/e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl"
Sep 14 10:05:19.133119 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:19.133083    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d80c24f-867f-4c36-9e8b-3ab262ef1fb7-workload-socket\") pod \"e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl\" (UID: \"3d80c24f-867f-4c36-9e8b-3ab262ef1fb7\") " pod="openshift-ingress/e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl"
Sep 14 10:05:19.133334 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:19.133308    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d80c24f-867f-4c36-9e8b-3ab262ef1fb7-istiod-ca-cert\") pod \"e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl\" (UID: \"3d80c24f-867f-4c36-9e8b-3ab262ef1fb7\") " pod="openshift-ingress/e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl"
Sep 14 10:05:19.133468 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:19.133447    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d80c24f-867f-4c36-9e8b-3ab262ef1fb7-workload-certs\") pod \"e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl\" (UID: \"3d80c24f-867f-4c36-9e8b-3ab262ef1fb7\") " pod="openshift-ingress/e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl"
Sep 14 10:05:19.133556 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:19.133532    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d80c24f-867f-4c36-9e8b-3ab262ef1fb7-credential-socket\") pod \"e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl\" (UID: \"3d80c24f-867f-4c36-9e8b-3ab262ef1fb7\") " pod="openshift-ingress/e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl"
Sep 14 10:05:19.135113 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:19.135089    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d80c24f-867f-4c36-9e8b-3ab262ef1fb7-istio-envoy\") pod \"e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl\" (UID: \"3d80c24f-867f-4c36-9e8b-3ab262ef1fb7\") " pod="openshift-ingress/e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl"
Sep 14 10:05:19.135677 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:19.135656    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d80c24f-867f-4c36-9e8b-3ab262ef1fb7-istio-podinfo\") pod \"e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl\" (UID: \"3d80c24f-867f-4c36-9e8b-3ab262ef1fb7\") " pod="openshift-ingress/e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl"
Sep 14 10:05:19.143462 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:19.143429    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d80c24f-867f-4c36-9e8b-3ab262ef1fb7-istio-token\") pod \"e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl\" (UID: \"3d80c24f-867f-4c36-9e8b-3ab262ef1fb7\") " pod="openshift-ingress/e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl"
Sep 14 10:05:19.144245 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:19.144216    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29szt\" (UniqueName: \"kubernetes.io/projected/3d80c24f-867f-4c36-9e8b-3ab262ef1fb7-kube-api-access-29szt\") pod \"e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl\" (UID: \"3d80c24f-867f-4c36-9e8b-3ab262ef1fb7\") " pod="openshift-ingress/e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl"
Sep 14 10:05:19.272448 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:19.272410    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl"
Sep 14 10:05:19.341431 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:19.341389    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:05:19.446357 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:19.446303    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:05:19.492694 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:19.492657    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp" event={"ID":"e1bb4690-068b-43c3-abdb-8bcda1591ae4","Type":"ContainerStarted","Data":"db6fbf4ee39a7355080a64b101402f12d2fcf8c529d7b5256ba8b0d40ef5d2a5"}
Sep 14 10:05:19.590101 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:19.590032    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl"]
Sep 14 10:05:19.592568 ip-10-0-132-4 kubenswrapper[2563]: W0914 10:05:19.592530    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d80c24f_867f_4c36_9e8b_3ab262ef1fb7.slice/crio-c9675e4b528abddf1c8692f46b8525688d555be4466eab5594b18b7cd6b43e2d WatchSource:0}: Error finding container c9675e4b528abddf1c8692f46b8525688d555be4466eab5594b18b7cd6b43e2d: Status 404 returned error can't find the container with id c9675e4b528abddf1c8692f46b8525688d555be4466eab5594b18b7cd6b43e2d
Sep 14 10:05:19.595151 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:19.595113    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 10:05:19.595265 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:19.595197    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 10:05:19.595265 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:19.595248    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 10:05:20.285259 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:20.285216    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp"
Sep 14 10:05:20.285696 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:20.285477    2563 patch_prober.go:28] interesting pod/e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Sep 14 10:05:20.285696 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:20.285521    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp" podUID="e1bb4690-068b-43c3-abdb-8bcda1591ae4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Sep 14 10:05:20.402778 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:20.402734    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:05:20.501112 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:20.501072    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl" event={"ID":"3d80c24f-867f-4c36-9e8b-3ab262ef1fb7","Type":"ContainerStarted","Data":"cf1ead24add79491bffa8657c0af3e606ddd578d947b41d76939f11b5f9ff21c"}
Sep 14 10:05:20.501299 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:20.501132    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl" event={"ID":"3d80c24f-867f-4c36-9e8b-3ab262ef1fb7","Type":"ContainerStarted","Data":"c9675e4b528abddf1c8692f46b8525688d555be4466eab5594b18b7cd6b43e2d"}
Sep 14 10:05:20.534517 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:20.534431    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl" podStartSLOduration=2.534405449 podStartE2EDuration="2.534405449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:05:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:05:20.532189476 +0000 UTC m=+1309.571864098" watchObservedRunningTime="2026-09-14 10:05:20.534405449 +0000 UTC m=+1309.574080093"
Sep 14 10:05:21.265476 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:21.265439    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:05:21.273641 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:21.273610    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl"
Sep 14 10:05:21.279446 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:21.279414    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl"
Sep 14 10:05:21.284983 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:21.284951    2563 patch_prober.go:28] interesting pod/e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Sep 14 10:05:21.285133 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:21.285033    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp" podUID="e1bb4690-068b-43c3-abdb-8bcda1591ae4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Sep 14 10:05:21.508699 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:21.508660    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl"
Sep 14 10:05:21.509688 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:21.509670    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl"
Sep 14 10:05:22.285241 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:22.285203    2563 patch_prober.go:28] interesting pod/e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Sep 14 10:05:22.285439 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:22.285264    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp" podUID="e1bb4690-068b-43c3-abdb-8bcda1591ae4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Sep 14 10:05:22.396044 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:22.395976    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:05:22.561487 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:22.561388    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:05:22.662208 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:22.662174    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:05:23.284728 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:23.284690    2563 patch_prober.go:28] interesting pod/e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Sep 14 10:05:23.284927 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:23.284747    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp" podUID="e1bb4690-068b-43c3-abdb-8bcda1591ae4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Sep 14 10:05:23.335677 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:23.335629    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:05:24.220705 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:24.220667    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:05:24.284961 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:24.284916    2563 patch_prober.go:28] interesting pod/e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Sep 14 10:05:24.285191 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:24.284991    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp" podUID="e1bb4690-068b-43c3-abdb-8bcda1591ae4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Sep 14 10:05:25.140954 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:25.140918    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:05:25.290462 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:25.290416    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp"
Sep 14 10:05:25.290891 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:25.290864    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp"
Sep 14 10:05:25.293468 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:25.293440    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp"
Sep 14 10:05:25.876650 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:25.876614    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:05:26.779777 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:26.779742    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:05:27.445325 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:27.445284    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-2901ec/distinct-w4-2901ec-kserve-9fc4b5697-xblp8"]
Sep 14 10:05:27.448078 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:27.448037    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2901ec/distinct-w4-2901ec-kserve-9fc4b5697-xblp8"
Sep 14 10:05:27.450235 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:27.450205    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-2901ec\"/\"distinct-w4-2901ec-kserve-self-signed-certs\""
Sep 14 10:05:27.459708 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:27.459676    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-2901ec/distinct-w4-2901ec-kserve-9fc4b5697-xblp8"]
Sep 14 10:05:27.470889 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:27.470853    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:05:27.532106 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:27.532057    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc-tmp-dir\") pod \"distinct-w4-2901ec-kserve-9fc4b5697-xblp8\" (UID: \"3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc\") " pod="e2e-models-e2e-worker-w4-2901ec/distinct-w4-2901ec-kserve-9fc4b5697-xblp8"
Sep 14 10:05:27.532389 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:27.532362    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vfk6\" (UniqueName: \"kubernetes.io/projected/3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc-kube-api-access-4vfk6\") pod \"distinct-w4-2901ec-kserve-9fc4b5697-xblp8\" (UID: \"3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc\") " pod="e2e-models-e2e-worker-w4-2901ec/distinct-w4-2901ec-kserve-9fc4b5697-xblp8"
Sep 14 10:05:27.532533 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:27.532402    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc-dshm\") pod \"distinct-w4-2901ec-kserve-9fc4b5697-xblp8\" (UID: \"3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc\") " pod="e2e-models-e2e-worker-w4-2901ec/distinct-w4-2901ec-kserve-9fc4b5697-xblp8"
Sep 14 10:05:27.532533 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:27.532497    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc-home\") pod \"distinct-w4-2901ec-kserve-9fc4b5697-xblp8\" (UID: \"3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc\") " pod="e2e-models-e2e-worker-w4-2901ec/distinct-w4-2901ec-kserve-9fc4b5697-xblp8"
Sep 14 10:05:27.532657 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:27.532600    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc-tls-certs\") pod \"distinct-w4-2901ec-kserve-9fc4b5697-xblp8\" (UID: \"3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc\") " pod="e2e-models-e2e-worker-w4-2901ec/distinct-w4-2901ec-kserve-9fc4b5697-xblp8"
Sep 14 10:05:27.532700 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:27.532660    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc-model-cache\") pod \"distinct-w4-2901ec-kserve-9fc4b5697-xblp8\" (UID: \"3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc\") " pod="e2e-models-e2e-worker-w4-2901ec/distinct-w4-2901ec-kserve-9fc4b5697-xblp8"
Sep 14 10:05:27.571790 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:27.571758    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:05:27.638034 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:27.634446    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc-tls-certs\") pod \"distinct-w4-2901ec-kserve-9fc4b5697-xblp8\" (UID: \"3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc\") " pod="e2e-models-e2e-worker-w4-2901ec/distinct-w4-2901ec-kserve-9fc4b5697-xblp8"
Sep 14 10:05:27.638034 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:27.634547    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc-model-cache\") pod \"distinct-w4-2901ec-kserve-9fc4b5697-xblp8\" (UID: \"3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc\") " pod="e2e-models-e2e-worker-w4-2901ec/distinct-w4-2901ec-kserve-9fc4b5697-xblp8"
Sep 14 10:05:27.638034 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:27.634680    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc-tmp-dir\") pod \"distinct-w4-2901ec-kserve-9fc4b5697-xblp8\" (UID: \"3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc\") " pod="e2e-models-e2e-worker-w4-2901ec/distinct-w4-2901ec-kserve-9fc4b5697-xblp8"
Sep 14 10:05:27.638034 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:27.634713    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vfk6\" (UniqueName: \"kubernetes.io/projected/3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc-kube-api-access-4vfk6\") pod \"distinct-w4-2901ec-kserve-9fc4b5697-xblp8\" (UID: \"3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc\") " pod="e2e-models-e2e-worker-w4-2901ec/distinct-w4-2901ec-kserve-9fc4b5697-xblp8"
Sep 14 10:05:27.638034 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:27.634744    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc-dshm\") pod \"distinct-w4-2901ec-kserve-9fc4b5697-xblp8\" (UID: \"3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc\") " pod="e2e-models-e2e-worker-w4-2901ec/distinct-w4-2901ec-kserve-9fc4b5697-xblp8"
Sep 14 10:05:27.638034 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:27.634790    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc-home\") pod \"distinct-w4-2901ec-kserve-9fc4b5697-xblp8\" (UID: \"3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc\") " pod="e2e-models-e2e-worker-w4-2901ec/distinct-w4-2901ec-kserve-9fc4b5697-xblp8"
Sep 14 10:05:27.638034 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:27.635116    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc-model-cache\") pod \"distinct-w4-2901ec-kserve-9fc4b5697-xblp8\" (UID: \"3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc\") " pod="e2e-models-e2e-worker-w4-2901ec/distinct-w4-2901ec-kserve-9fc4b5697-xblp8"
Sep 14 10:05:27.638034 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:27.635173    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc-home\") pod \"distinct-w4-2901ec-kserve-9fc4b5697-xblp8\" (UID: \"3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc\") " pod="e2e-models-e2e-worker-w4-2901ec/distinct-w4-2901ec-kserve-9fc4b5697-xblp8"
Sep 14 10:05:27.638034 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:27.635638    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc-tmp-dir\") pod \"distinct-w4-2901ec-kserve-9fc4b5697-xblp8\" (UID: \"3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc\") " pod="e2e-models-e2e-worker-w4-2901ec/distinct-w4-2901ec-kserve-9fc4b5697-xblp8"
Sep 14 10:05:27.638659 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:27.638380    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc-tls-certs\") pod \"distinct-w4-2901ec-kserve-9fc4b5697-xblp8\" (UID: \"3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc\") " pod="e2e-models-e2e-worker-w4-2901ec/distinct-w4-2901ec-kserve-9fc4b5697-xblp8"
Sep 14 10:05:27.639028 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:27.638849    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc-dshm\") pod \"distinct-w4-2901ec-kserve-9fc4b5697-xblp8\" (UID: \"3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc\") " pod="e2e-models-e2e-worker-w4-2901ec/distinct-w4-2901ec-kserve-9fc4b5697-xblp8"
Sep 14 10:05:27.644295 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:27.644221    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vfk6\" (UniqueName: \"kubernetes.io/projected/3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc-kube-api-access-4vfk6\") pod \"distinct-w4-2901ec-kserve-9fc4b5697-xblp8\" (UID: \"3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc\") " pod="e2e-models-e2e-worker-w4-2901ec/distinct-w4-2901ec-kserve-9fc4b5697-xblp8"
Sep 14 10:05:27.759603 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:27.759559    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2901ec/distinct-w4-2901ec-kserve-9fc4b5697-xblp8"
Sep 14 10:05:28.039528 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:28.039497    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-2901ec/distinct-w4-2901ec-kserve-9fc4b5697-xblp8"]
Sep 14 10:05:28.041671 ip-10-0-132-4 kubenswrapper[2563]: W0914 10:05:28.041637    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c15cf83_c6b9_4371_8e1e_3d5c3f6d1fbc.slice/crio-a7ddfd69422e2f21fd7cacce95aca3083ac0ee91cf93d791c7cc6bdc6bfc613e WatchSource:0}: Error finding container a7ddfd69422e2f21fd7cacce95aca3083ac0ee91cf93d791c7cc6bdc6bfc613e: Status 404 returned error can't find the container with id a7ddfd69422e2f21fd7cacce95aca3083ac0ee91cf93d791c7cc6bdc6bfc613e
Sep 14 10:05:28.421666 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:28.421570    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:05:28.520412 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:28.520371    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:05:28.574889 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:28.574748    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2901ec/distinct-w4-2901ec-kserve-9fc4b5697-xblp8" event={"ID":"3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc","Type":"ContainerStarted","Data":"f398d11a46bcde229405a18228763a69badb0d7b2bf6da64c7ef895d363c3ec9"}
Sep 14 10:05:28.574889 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:28.574805    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2901ec/distinct-w4-2901ec-kserve-9fc4b5697-xblp8" event={"ID":"3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc","Type":"ContainerStarted","Data":"a7ddfd69422e2f21fd7cacce95aca3083ac0ee91cf93d791c7cc6bdc6bfc613e"}
Sep 14 10:05:28.593878 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:28.593814    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-2901ec/distinct-w4-2901ec-kserve-9fc4b5697-xblp8" podStartSLOduration=1.593791945 podStartE2EDuration="1.593791945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:05:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:05:28.591577638 +0000 UTC m=+1317.631252261" watchObservedRunningTime="2026-09-14 10:05:28.593791945 +0000 UTC m=+1317.633466600"
Sep 14 10:05:28.674300 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:28.674202    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:05:29.034158 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:29.034123    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:05:29.135637 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:29.135592    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:05:31.682722 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:31.682686    2563 scope.go:117] "RemoveContainer" containerID="98e61729d23719c4c2922bac670fe143f17e5be70b98a591ae4ac2d04e8c77b5"
Sep 14 10:05:37.760468 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:37.760425    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-2901ec/distinct-w4-2901ec-kserve-9fc4b5697-xblp8"
Sep 14 10:05:37.760879 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:37.760492    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2901ec/distinct-w4-2901ec-kserve-9fc4b5697-xblp8"
Sep 14 10:05:37.773913 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:37.773882    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-2901ec/distinct-w4-2901ec-kserve-9fc4b5697-xblp8"
Sep 14 10:05:38.652657 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:38.652621    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2901ec/distinct-w4-2901ec-kserve-9fc4b5697-xblp8"
Sep 14 10:05:40.956664 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:40.956616    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-75017ebc/e2e-lifecycle-model-75017ebc-kserve-c88f6db99-29nzt"]
Sep 14 10:05:40.960854 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:40.960816    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-75017ebc/e2e-lifecycle-model-75017ebc-kserve-c88f6db99-29nzt"
Sep 14 10:05:40.963888 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:40.963843    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-75017ebc\"/\"default-dockercfg-nnmbw\""
Sep 14 10:05:40.964114 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:40.963971    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-75017ebc\"/\"kube-root-ca.crt\""
Sep 14 10:05:40.964114 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:40.964106    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-75017ebc\"/\"openshift-service-ca.crt\""
Sep 14 10:05:40.964262 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:40.964116    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-75017ebc\"/\"e2e-lifecycle-model-75017ebc-kserve-self-signed-certs\""
Sep 14 10:05:40.976036 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:40.975984    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-75017ebc/e2e-lifecycle-model-75017ebc-kserve-c88f6db99-29nzt"]
Sep 14 10:05:40.976364 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:40.976342    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/becc30c1-8ba3-4252-99d8-6a2c55441916-tls-certs\") pod \"e2e-lifecycle-model-75017ebc-kserve-c88f6db99-29nzt\" (UID: \"becc30c1-8ba3-4252-99d8-6a2c55441916\") " pod="ai-tenant-e2e-mt-75017ebc/e2e-lifecycle-model-75017ebc-kserve-c88f6db99-29nzt"
Sep 14 10:05:40.976452 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:40.976379    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/becc30c1-8ba3-4252-99d8-6a2c55441916-home\") pod \"e2e-lifecycle-model-75017ebc-kserve-c88f6db99-29nzt\" (UID: \"becc30c1-8ba3-4252-99d8-6a2c55441916\") " pod="ai-tenant-e2e-mt-75017ebc/e2e-lifecycle-model-75017ebc-kserve-c88f6db99-29nzt"
Sep 14 10:05:40.976518 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:40.976466    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/becc30c1-8ba3-4252-99d8-6a2c55441916-tmp-dir\") pod \"e2e-lifecycle-model-75017ebc-kserve-c88f6db99-29nzt\" (UID: \"becc30c1-8ba3-4252-99d8-6a2c55441916\") " pod="ai-tenant-e2e-mt-75017ebc/e2e-lifecycle-model-75017ebc-kserve-c88f6db99-29nzt"
Sep 14 10:05:40.976518 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:40.976498    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5wjq\" (UniqueName: \"kubernetes.io/projected/becc30c1-8ba3-4252-99d8-6a2c55441916-kube-api-access-z5wjq\") pod \"e2e-lifecycle-model-75017ebc-kserve-c88f6db99-29nzt\" (UID: \"becc30c1-8ba3-4252-99d8-6a2c55441916\") " pod="ai-tenant-e2e-mt-75017ebc/e2e-lifecycle-model-75017ebc-kserve-c88f6db99-29nzt"
Sep 14 10:05:40.976604 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:40.976531    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/becc30c1-8ba3-4252-99d8-6a2c55441916-dshm\") pod \"e2e-lifecycle-model-75017ebc-kserve-c88f6db99-29nzt\" (UID: \"becc30c1-8ba3-4252-99d8-6a2c55441916\") " pod="ai-tenant-e2e-mt-75017ebc/e2e-lifecycle-model-75017ebc-kserve-c88f6db99-29nzt"
Sep 14 10:05:40.976604 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:40.976581    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/becc30c1-8ba3-4252-99d8-6a2c55441916-model-cache\") pod \"e2e-lifecycle-model-75017ebc-kserve-c88f6db99-29nzt\" (UID: \"becc30c1-8ba3-4252-99d8-6a2c55441916\") " pod="ai-tenant-e2e-mt-75017ebc/e2e-lifecycle-model-75017ebc-kserve-c88f6db99-29nzt"
Sep 14 10:05:40.992337 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:40.992299    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:05:41.078039 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:41.077972    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/becc30c1-8ba3-4252-99d8-6a2c55441916-model-cache\") pod \"e2e-lifecycle-model-75017ebc-kserve-c88f6db99-29nzt\" (UID: \"becc30c1-8ba3-4252-99d8-6a2c55441916\") " pod="ai-tenant-e2e-mt-75017ebc/e2e-lifecycle-model-75017ebc-kserve-c88f6db99-29nzt"
Sep 14 10:05:41.078230 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:41.078130    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/becc30c1-8ba3-4252-99d8-6a2c55441916-tls-certs\") pod \"e2e-lifecycle-model-75017ebc-kserve-c88f6db99-29nzt\" (UID: \"becc30c1-8ba3-4252-99d8-6a2c55441916\") " pod="ai-tenant-e2e-mt-75017ebc/e2e-lifecycle-model-75017ebc-kserve-c88f6db99-29nzt"
Sep 14 10:05:41.078230 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:41.078166    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/becc30c1-8ba3-4252-99d8-6a2c55441916-home\") pod \"e2e-lifecycle-model-75017ebc-kserve-c88f6db99-29nzt\" (UID: \"becc30c1-8ba3-4252-99d8-6a2c55441916\") " pod="ai-tenant-e2e-mt-75017ebc/e2e-lifecycle-model-75017ebc-kserve-c88f6db99-29nzt"
Sep 14 10:05:41.078351 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:41.078242    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/becc30c1-8ba3-4252-99d8-6a2c55441916-tmp-dir\") pod \"e2e-lifecycle-model-75017ebc-kserve-c88f6db99-29nzt\" (UID: \"becc30c1-8ba3-4252-99d8-6a2c55441916\") " pod="ai-tenant-e2e-mt-75017ebc/e2e-lifecycle-model-75017ebc-kserve-c88f6db99-29nzt"
Sep 14 10:05:41.078351 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:41.078270    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5wjq\" (UniqueName: \"kubernetes.io/projected/becc30c1-8ba3-4252-99d8-6a2c55441916-kube-api-access-z5wjq\") pod \"e2e-lifecycle-model-75017ebc-kserve-c88f6db99-29nzt\" (UID: \"becc30c1-8ba3-4252-99d8-6a2c55441916\") " pod="ai-tenant-e2e-mt-75017ebc/e2e-lifecycle-model-75017ebc-kserve-c88f6db99-29nzt"
Sep 14 10:05:41.078457 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:41.078358    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/becc30c1-8ba3-4252-99d8-6a2c55441916-dshm\") pod \"e2e-lifecycle-model-75017ebc-kserve-c88f6db99-29nzt\" (UID: \"becc30c1-8ba3-4252-99d8-6a2c55441916\") " pod="ai-tenant-e2e-mt-75017ebc/e2e-lifecycle-model-75017ebc-kserve-c88f6db99-29nzt"
Sep 14 10:05:41.078507 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:41.078468    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/becc30c1-8ba3-4252-99d8-6a2c55441916-model-cache\") pod \"e2e-lifecycle-model-75017ebc-kserve-c88f6db99-29nzt\" (UID: \"becc30c1-8ba3-4252-99d8-6a2c55441916\") " pod="ai-tenant-e2e-mt-75017ebc/e2e-lifecycle-model-75017ebc-kserve-c88f6db99-29nzt"
Sep 14 10:05:41.078606 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:41.078582    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/becc30c1-8ba3-4252-99d8-6a2c55441916-tmp-dir\") pod \"e2e-lifecycle-model-75017ebc-kserve-c88f6db99-29nzt\" (UID: \"becc30c1-8ba3-4252-99d8-6a2c55441916\") " pod="ai-tenant-e2e-mt-75017ebc/e2e-lifecycle-model-75017ebc-kserve-c88f6db99-29nzt"
Sep 14 10:05:41.078660 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:41.078619    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/becc30c1-8ba3-4252-99d8-6a2c55441916-home\") pod \"e2e-lifecycle-model-75017ebc-kserve-c88f6db99-29nzt\" (UID: \"becc30c1-8ba3-4252-99d8-6a2c55441916\") " pod="ai-tenant-e2e-mt-75017ebc/e2e-lifecycle-model-75017ebc-kserve-c88f6db99-29nzt"
Sep 14 10:05:41.080662 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:41.080633    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/becc30c1-8ba3-4252-99d8-6a2c55441916-dshm\") pod \"e2e-lifecycle-model-75017ebc-kserve-c88f6db99-29nzt\" (UID: \"becc30c1-8ba3-4252-99d8-6a2c55441916\") " pod="ai-tenant-e2e-mt-75017ebc/e2e-lifecycle-model-75017ebc-kserve-c88f6db99-29nzt"
Sep 14 10:05:41.080924 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:41.080905    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/becc30c1-8ba3-4252-99d8-6a2c55441916-tls-certs\") pod \"e2e-lifecycle-model-75017ebc-kserve-c88f6db99-29nzt\" (UID: \"becc30c1-8ba3-4252-99d8-6a2c55441916\") " pod="ai-tenant-e2e-mt-75017ebc/e2e-lifecycle-model-75017ebc-kserve-c88f6db99-29nzt"
Sep 14 10:05:41.087919 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:41.087883    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5wjq\" (UniqueName: \"kubernetes.io/projected/becc30c1-8ba3-4252-99d8-6a2c55441916-kube-api-access-z5wjq\") pod \"e2e-lifecycle-model-75017ebc-kserve-c88f6db99-29nzt\" (UID: \"becc30c1-8ba3-4252-99d8-6a2c55441916\") " pod="ai-tenant-e2e-mt-75017ebc/e2e-lifecycle-model-75017ebc-kserve-c88f6db99-29nzt"
Sep 14 10:05:41.275163 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:41.275125    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-75017ebc/e2e-lifecycle-model-75017ebc-kserve-c88f6db99-29nzt"
Sep 14 10:05:41.552648 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:41.552616    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-75017ebc/e2e-lifecycle-model-75017ebc-kserve-c88f6db99-29nzt"]
Sep 14 10:05:41.554560 ip-10-0-132-4 kubenswrapper[2563]: W0914 10:05:41.554530    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbecc30c1_8ba3_4252_99d8_6a2c55441916.slice/crio-1ed19d1f019245b4a6a5412b4d7606fe900b0938e6d65dd226c0c02035ca9ca7 WatchSource:0}: Error finding container 1ed19d1f019245b4a6a5412b4d7606fe900b0938e6d65dd226c0c02035ca9ca7: Status 404 returned error can't find the container with id 1ed19d1f019245b4a6a5412b4d7606fe900b0938e6d65dd226c0c02035ca9ca7
Sep 14 10:05:41.662418 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:41.662380    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-75017ebc/e2e-lifecycle-model-75017ebc-kserve-c88f6db99-29nzt" event={"ID":"becc30c1-8ba3-4252-99d8-6a2c55441916","Type":"ContainerStarted","Data":"f2a671c0b43416d95edce0330f96e7f9b4612de99f6f0b3b2e9035448b76c5ea"}
Sep 14 10:05:41.662599 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:41.662427    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-75017ebc/e2e-lifecycle-model-75017ebc-kserve-c88f6db99-29nzt" event={"ID":"becc30c1-8ba3-4252-99d8-6a2c55441916","Type":"ContainerStarted","Data":"1ed19d1f019245b4a6a5412b4d7606fe900b0938e6d65dd226c0c02035ca9ca7"}
Sep 14 10:05:41.680839 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:41.680762    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-75017ebc/e2e-lifecycle-model-75017ebc-kserve-c88f6db99-29nzt" podStartSLOduration=1.680738243 podStartE2EDuration="1.680738243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:05:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:05:41.679642764 +0000 UTC m=+1330.719317390" watchObservedRunningTime="2026-09-14 10:05:41.680738243 +0000 UTC m=+1330.720412865"
Sep 14 10:05:50.990386 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:50.990342    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:05:51.275387 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:51.275340    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-75017ebc/e2e-lifecycle-model-75017ebc-kserve-c88f6db99-29nzt"
Sep 14 10:05:51.275612 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:51.275402    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-75017ebc/e2e-lifecycle-model-75017ebc-kserve-c88f6db99-29nzt"
Sep 14 10:05:51.288483 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:51.288451    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-75017ebc/e2e-lifecycle-model-75017ebc-kserve-c88f6db99-29nzt"
Sep 14 10:05:51.727518 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:51.727402    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:05:51.751139 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:51.751103    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-75017ebc/e2e-lifecycle-model-75017ebc-kserve-c88f6db99-29nzt"
Sep 14 10:05:52.488176 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:52.488131    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:05:52.551265 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:52.551224    2563 patch_prober.go:28] interesting pod/e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 14 10:05:52.551446 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:52.551309    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl" podUID="3d80c24f-867f-4c36-9e8b-3ab262ef1fb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 14 10:05:52.588295 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:52.588244    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:05:53.425248 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:53.425151    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:05:54.201881 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:54.201842    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:05:54.759208 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:54.759172    2563 generic.go:356] "Generic (PLEG): container finished" podID="3d80c24f-867f-4c36-9e8b-3ab262ef1fb7" containerID="cf1ead24add79491bffa8657c0af3e606ddd578d947b41d76939f11b5f9ff21c" exitCode=0
Sep 14 10:05:54.759434 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:54.759244    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl" event={"ID":"3d80c24f-867f-4c36-9e8b-3ab262ef1fb7","Type":"ContainerDied","Data":"cf1ead24add79491bffa8657c0af3e606ddd578d947b41d76939f11b5f9ff21c"}
Sep 14 10:05:54.759745 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:54.759726    2563 scope.go:117] "RemoveContainer" containerID="cf1ead24add79491bffa8657c0af3e606ddd578d947b41d76939f11b5f9ff21c"
Sep 14 10:05:54.761185 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:54.761152    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 10:05:54.761262 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:54.761231    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 10:05:54.761309 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:54.761285    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 10:05:55.063074 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:55.062979    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:05:55.769246 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:55.769198    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl" event={"ID":"3d80c24f-867f-4c36-9e8b-3ab262ef1fb7","Type":"ContainerStarted","Data":"6404f8bf1af89d10c30c8c85d005419ef6e72c50ef56a42c9636d841c9dd3813"}
Sep 14 10:05:56.008239 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:56.008199    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:05:56.273033 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:56.272958    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl"
Sep 14 10:05:56.273290 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:56.273263    2563 patch_prober.go:28] interesting pod/e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Sep 14 10:05:56.273345 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:56.273319    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl" podUID="3d80c24f-867f-4c36-9e8b-3ab262ef1fb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Sep 14 10:05:56.836351 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:56.836312    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:05:57.272879 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:57.272837    2563 patch_prober.go:28] interesting pod/e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Sep 14 10:05:57.273117 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:57.272903    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl" podUID="3d80c24f-867f-4c36-9e8b-3ab262ef1fb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Sep 14 10:05:58.273600 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:58.273563    2563 patch_prober.go:28] interesting pod/e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Sep 14 10:05:58.274073 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:58.273618    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl" podUID="3d80c24f-867f-4c36-9e8b-3ab262ef1fb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Sep 14 10:05:59.272799 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:59.272754    2563 patch_prober.go:28] interesting pod/e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Sep 14 10:05:59.272992 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:05:59.272811    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl" podUID="3d80c24f-867f-4c36-9e8b-3ab262ef1fb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Sep 14 10:06:00.273561 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:00.273518    2563 patch_prober.go:28] interesting pod/e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Sep 14 10:06:00.273985 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:00.273582    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl" podUID="3d80c24f-867f-4c36-9e8b-3ab262ef1fb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Sep 14 10:06:01.298116 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:01.298056    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl"
Sep 14 10:06:01.298661 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:01.298463    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl"
Sep 14 10:06:01.299437 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:01.299390    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl"
Sep 14 10:06:31.714336 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:31.714297    2563 scope.go:117] "RemoveContainer" containerID="81530ebfa06370b4e74fdefb5fb6d9acb28b651ba4c15d4fa80ec6bbf74d1708"
Sep 14 10:06:31.724196 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:31.724171    2563 scope.go:117] "RemoveContainer" containerID="0c35c8a327c3b23125bbdb1bc4f199afbc2cf72633c32d105b5714f6101d068f"
Sep 14 10:06:31.733906 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:31.733878    2563 scope.go:117] "RemoveContainer" containerID="ce2d1a938076f6e45916cfe1d9dd46da4c8fb0cab318181af11d18a37cefa812"
Sep 14 10:06:33.419670 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:33.419625    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-75017ebc-q7s8n"]
Sep 14 10:06:33.423205 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:33.423178    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-75017ebc-q7s8n"
Sep 14 10:06:33.425277 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:33.425248    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gf4xj\""
Sep 14 10:06:33.425277 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:33.425248    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 14 10:06:33.437696 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:33.437661    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-75017ebc-q7s8n"]
Sep 14 10:06:33.581558 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:33.581503    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/817ca3cc-631b-40a4-bf2d-ce2e77dd3e24-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-75017ebc-q7s8n\" (UID: \"817ca3cc-631b-40a4-bf2d-ce2e77dd3e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-75017ebc-q7s8n"
Sep 14 10:06:33.595150 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:33.595118    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:06:33.682738 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:33.682631    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/817ca3cc-631b-40a4-bf2d-ce2e77dd3e24-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-75017ebc-q7s8n\" (UID: \"817ca3cc-631b-40a4-bf2d-ce2e77dd3e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-75017ebc-q7s8n"
Sep 14 10:06:33.683410 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:33.683387    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/817ca3cc-631b-40a4-bf2d-ce2e77dd3e24-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-75017ebc-q7s8n\" (UID: \"817ca3cc-631b-40a4-bf2d-ce2e77dd3e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-75017ebc-q7s8n"
Sep 14 10:06:33.734770 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:33.734731    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-75017ebc-q7s8n"
Sep 14 10:06:34.017395 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:34.017362    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-75017ebc-q7s8n"]
Sep 14 10:06:34.018743 ip-10-0-132-4 kubenswrapper[2563]: W0914 10:06:34.018714    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod817ca3cc_631b_40a4_bf2d_ce2e77dd3e24.slice/crio-e8d333fcde11fc76b5842872001feaf3f290137ca74f6cdf226179cf578c9309 WatchSource:0}: Error finding container e8d333fcde11fc76b5842872001feaf3f290137ca74f6cdf226179cf578c9309: Status 404 returned error can't find the container with id e8d333fcde11fc76b5842872001feaf3f290137ca74f6cdf226179cf578c9309
Sep 14 10:06:34.488801 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:34.488768    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:06:35.035858 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:35.035817    2563 generic.go:356] "Generic (PLEG): container finished" podID="817ca3cc-631b-40a4-bf2d-ce2e77dd3e24" containerID="de66c10475da8f46b0d3406b641f44a0cec688d223d509e328db1555359797e4" exitCode=0
Sep 14 10:06:35.036036 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:35.035909    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-75017ebc-q7s8n" event={"ID":"817ca3cc-631b-40a4-bf2d-ce2e77dd3e24","Type":"ContainerDied","Data":"de66c10475da8f46b0d3406b641f44a0cec688d223d509e328db1555359797e4"}
Sep 14 10:06:35.036036 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:35.035972    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-75017ebc-q7s8n" event={"ID":"817ca3cc-631b-40a4-bf2d-ce2e77dd3e24","Type":"ContainerStarted","Data":"e8d333fcde11fc76b5842872001feaf3f290137ca74f6cdf226179cf578c9309"}
Sep 14 10:06:35.386896 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:35.386855    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:06:36.112756 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:36.112721    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:06:36.191805 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:36.191774    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-75017ebc-q7s8n"
Sep 14 10:06:36.308509 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:36.308405    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/817ca3cc-631b-40a4-bf2d-ce2e77dd3e24-maas-api-service-ca\") pod \"817ca3cc-631b-40a4-bf2d-ce2e77dd3e24\" (UID: \"817ca3cc-631b-40a4-bf2d-ce2e77dd3e24\") "
Sep 14 10:06:36.308816 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:36.308789    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/817ca3cc-631b-40a4-bf2d-ce2e77dd3e24-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "817ca3cc-631b-40a4-bf2d-ce2e77dd3e24" (UID: "817ca3cc-631b-40a4-bf2d-ce2e77dd3e24"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:06:36.410307 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:36.410261    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/817ca3cc-631b-40a4-bf2d-ce2e77dd3e24-maas-api-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:06:37.009641 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:37.009601    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:06:37.050715 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:37.050683    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-75017ebc-q7s8n"
Sep 14 10:06:37.050963 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:37.050723    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-75017ebc-q7s8n" event={"ID":"817ca3cc-631b-40a4-bf2d-ce2e77dd3e24","Type":"ContainerDied","Data":"e8d333fcde11fc76b5842872001feaf3f290137ca74f6cdf226179cf578c9309"}
Sep 14 10:06:37.050963 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:37.050782    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8d333fcde11fc76b5842872001feaf3f290137ca74f6cdf226179cf578c9309"
Sep 14 10:06:37.776565 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:37.776529    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:06:38.508991 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:38.508958    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:06:39.389472 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:39.389428    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:06:40.160268 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:40.160229    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:06:46.150094 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:46.150046    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-51aefd/auth-test-model-51aefd-kserve-7c9dbd4899-87r7z"]
Sep 14 10:06:46.150625 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:46.150473    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="817ca3cc-631b-40a4-bf2d-ce2e77dd3e24" containerName="revoke-keys"
Sep 14 10:06:46.150625 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:46.150485    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="817ca3cc-631b-40a4-bf2d-ce2e77dd3e24" containerName="revoke-keys"
Sep 14 10:06:46.150625 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:46.150562    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="817ca3cc-631b-40a4-bf2d-ce2e77dd3e24" containerName="revoke-keys"
Sep 14 10:06:46.152809 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:46.152779    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-51aefd/auth-test-model-51aefd-kserve-7c9dbd4899-87r7z"
Sep 14 10:06:46.155522 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:46.155488    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-51aefd\"/\"default-dockercfg-bbdgq\""
Sep 14 10:06:46.155522 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:46.155513    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-51aefd\"/\"openshift-service-ca.crt\""
Sep 14 10:06:46.155753 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:46.155488    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-51aefd\"/\"auth-test-model-51aefd-kserve-self-signed-certs\""
Sep 14 10:06:46.155753 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:46.155487    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-51aefd\"/\"kube-root-ca.crt\""
Sep 14 10:06:46.165052 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:46.164985    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-51aefd/auth-test-model-51aefd-kserve-7c9dbd4899-87r7z"]
Sep 14 10:06:46.185068 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:46.185024    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:06:46.212900 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:46.212855    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43d9ca79-ce89-424a-bb1b-dd8197cd5d6b-home\") pod \"auth-test-model-51aefd-kserve-7c9dbd4899-87r7z\" (UID: \"43d9ca79-ce89-424a-bb1b-dd8197cd5d6b\") " pod="ai-tenant-e2e-shared-a-w3-51aefd/auth-test-model-51aefd-kserve-7c9dbd4899-87r7z"
Sep 14 10:06:46.213125 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:46.212904    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43d9ca79-ce89-424a-bb1b-dd8197cd5d6b-dshm\") pod \"auth-test-model-51aefd-kserve-7c9dbd4899-87r7z\" (UID: \"43d9ca79-ce89-424a-bb1b-dd8197cd5d6b\") " pod="ai-tenant-e2e-shared-a-w3-51aefd/auth-test-model-51aefd-kserve-7c9dbd4899-87r7z"
Sep 14 10:06:46.213125 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:46.212929    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43d9ca79-ce89-424a-bb1b-dd8197cd5d6b-tls-certs\") pod \"auth-test-model-51aefd-kserve-7c9dbd4899-87r7z\" (UID: \"43d9ca79-ce89-424a-bb1b-dd8197cd5d6b\") " pod="ai-tenant-e2e-shared-a-w3-51aefd/auth-test-model-51aefd-kserve-7c9dbd4899-87r7z"
Sep 14 10:06:46.213125 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:46.212947    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43d9ca79-ce89-424a-bb1b-dd8197cd5d6b-tmp-dir\") pod \"auth-test-model-51aefd-kserve-7c9dbd4899-87r7z\" (UID: \"43d9ca79-ce89-424a-bb1b-dd8197cd5d6b\") " pod="ai-tenant-e2e-shared-a-w3-51aefd/auth-test-model-51aefd-kserve-7c9dbd4899-87r7z"
Sep 14 10:06:46.213125 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:46.213034    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43d9ca79-ce89-424a-bb1b-dd8197cd5d6b-model-cache\") pod \"auth-test-model-51aefd-kserve-7c9dbd4899-87r7z\" (UID: \"43d9ca79-ce89-424a-bb1b-dd8197cd5d6b\") " pod="ai-tenant-e2e-shared-a-w3-51aefd/auth-test-model-51aefd-kserve-7c9dbd4899-87r7z"
Sep 14 10:06:46.213272 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:46.213145    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bktws\" (UniqueName: \"kubernetes.io/projected/43d9ca79-ce89-424a-bb1b-dd8197cd5d6b-kube-api-access-bktws\") pod \"auth-test-model-51aefd-kserve-7c9dbd4899-87r7z\" (UID: \"43d9ca79-ce89-424a-bb1b-dd8197cd5d6b\") " pod="ai-tenant-e2e-shared-a-w3-51aefd/auth-test-model-51aefd-kserve-7c9dbd4899-87r7z"
Sep 14 10:06:46.314613 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:46.314571    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bktws\" (UniqueName: \"kubernetes.io/projected/43d9ca79-ce89-424a-bb1b-dd8197cd5d6b-kube-api-access-bktws\") pod \"auth-test-model-51aefd-kserve-7c9dbd4899-87r7z\" (UID: \"43d9ca79-ce89-424a-bb1b-dd8197cd5d6b\") " pod="ai-tenant-e2e-shared-a-w3-51aefd/auth-test-model-51aefd-kserve-7c9dbd4899-87r7z"
Sep 14 10:06:46.314797 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:46.314673    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43d9ca79-ce89-424a-bb1b-dd8197cd5d6b-home\") pod \"auth-test-model-51aefd-kserve-7c9dbd4899-87r7z\" (UID: \"43d9ca79-ce89-424a-bb1b-dd8197cd5d6b\") " pod="ai-tenant-e2e-shared-a-w3-51aefd/auth-test-model-51aefd-kserve-7c9dbd4899-87r7z"
Sep 14 10:06:46.314797 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:46.314699    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43d9ca79-ce89-424a-bb1b-dd8197cd5d6b-dshm\") pod \"auth-test-model-51aefd-kserve-7c9dbd4899-87r7z\" (UID: \"43d9ca79-ce89-424a-bb1b-dd8197cd5d6b\") " pod="ai-tenant-e2e-shared-a-w3-51aefd/auth-test-model-51aefd-kserve-7c9dbd4899-87r7z"
Sep 14 10:06:46.314797 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:46.314717    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43d9ca79-ce89-424a-bb1b-dd8197cd5d6b-tls-certs\") pod \"auth-test-model-51aefd-kserve-7c9dbd4899-87r7z\" (UID: \"43d9ca79-ce89-424a-bb1b-dd8197cd5d6b\") " pod="ai-tenant-e2e-shared-a-w3-51aefd/auth-test-model-51aefd-kserve-7c9dbd4899-87r7z"
Sep 14 10:06:46.314797 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:46.314736    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43d9ca79-ce89-424a-bb1b-dd8197cd5d6b-tmp-dir\") pod \"auth-test-model-51aefd-kserve-7c9dbd4899-87r7z\" (UID: \"43d9ca79-ce89-424a-bb1b-dd8197cd5d6b\") " pod="ai-tenant-e2e-shared-a-w3-51aefd/auth-test-model-51aefd-kserve-7c9dbd4899-87r7z"
Sep 14 10:06:46.314797 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:46.314761    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43d9ca79-ce89-424a-bb1b-dd8197cd5d6b-model-cache\") pod \"auth-test-model-51aefd-kserve-7c9dbd4899-87r7z\" (UID: \"43d9ca79-ce89-424a-bb1b-dd8197cd5d6b\") " pod="ai-tenant-e2e-shared-a-w3-51aefd/auth-test-model-51aefd-kserve-7c9dbd4899-87r7z"
Sep 14 10:06:46.315249 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:46.315214    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43d9ca79-ce89-424a-bb1b-dd8197cd5d6b-home\") pod \"auth-test-model-51aefd-kserve-7c9dbd4899-87r7z\" (UID: \"43d9ca79-ce89-424a-bb1b-dd8197cd5d6b\") " pod="ai-tenant-e2e-shared-a-w3-51aefd/auth-test-model-51aefd-kserve-7c9dbd4899-87r7z"
Sep 14 10:06:46.315352 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:46.315253    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43d9ca79-ce89-424a-bb1b-dd8197cd5d6b-model-cache\") pod \"auth-test-model-51aefd-kserve-7c9dbd4899-87r7z\" (UID: \"43d9ca79-ce89-424a-bb1b-dd8197cd5d6b\") " pod="ai-tenant-e2e-shared-a-w3-51aefd/auth-test-model-51aefd-kserve-7c9dbd4899-87r7z"
Sep 14 10:06:46.315352 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:46.315272    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43d9ca79-ce89-424a-bb1b-dd8197cd5d6b-tmp-dir\") pod \"auth-test-model-51aefd-kserve-7c9dbd4899-87r7z\" (UID: \"43d9ca79-ce89-424a-bb1b-dd8197cd5d6b\") " pod="ai-tenant-e2e-shared-a-w3-51aefd/auth-test-model-51aefd-kserve-7c9dbd4899-87r7z"
Sep 14 10:06:46.317169 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:46.317147    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43d9ca79-ce89-424a-bb1b-dd8197cd5d6b-dshm\") pod \"auth-test-model-51aefd-kserve-7c9dbd4899-87r7z\" (UID: \"43d9ca79-ce89-424a-bb1b-dd8197cd5d6b\") " pod="ai-tenant-e2e-shared-a-w3-51aefd/auth-test-model-51aefd-kserve-7c9dbd4899-87r7z"
Sep 14 10:06:46.317535 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:46.317513    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43d9ca79-ce89-424a-bb1b-dd8197cd5d6b-tls-certs\") pod \"auth-test-model-51aefd-kserve-7c9dbd4899-87r7z\" (UID: \"43d9ca79-ce89-424a-bb1b-dd8197cd5d6b\") " pod="ai-tenant-e2e-shared-a-w3-51aefd/auth-test-model-51aefd-kserve-7c9dbd4899-87r7z"
Sep 14 10:06:46.322619 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:46.322589    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bktws\" (UniqueName: \"kubernetes.io/projected/43d9ca79-ce89-424a-bb1b-dd8197cd5d6b-kube-api-access-bktws\") pod \"auth-test-model-51aefd-kserve-7c9dbd4899-87r7z\" (UID: \"43d9ca79-ce89-424a-bb1b-dd8197cd5d6b\") " pod="ai-tenant-e2e-shared-a-w3-51aefd/auth-test-model-51aefd-kserve-7c9dbd4899-87r7z"
Sep 14 10:06:46.465644 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:46.465537    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-51aefd/auth-test-model-51aefd-kserve-7c9dbd4899-87r7z"
Sep 14 10:06:46.746900 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:46.746863    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-51aefd/auth-test-model-51aefd-kserve-7c9dbd4899-87r7z"]
Sep 14 10:06:46.748516 ip-10-0-132-4 kubenswrapper[2563]: W0914 10:06:46.748481    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod43d9ca79_ce89_424a_bb1b_dd8197cd5d6b.slice/crio-74344fc177ab50bc952fa901949e3b440f25cf9fa156faee80407bcace503ff6 WatchSource:0}: Error finding container 74344fc177ab50bc952fa901949e3b440f25cf9fa156faee80407bcace503ff6: Status 404 returned error can't find the container with id 74344fc177ab50bc952fa901949e3b440f25cf9fa156faee80407bcace503ff6
Sep 14 10:06:47.116662 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:47.116559    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-51aefd/auth-test-model-51aefd-kserve-7c9dbd4899-87r7z" event={"ID":"43d9ca79-ce89-424a-bb1b-dd8197cd5d6b","Type":"ContainerStarted","Data":"515ce9622fc00fee569d9b68bcbe35504468ba0eb990f127c0de3aca4139c123"}
Sep 14 10:06:47.116662 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:47.116605    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-51aefd/auth-test-model-51aefd-kserve-7c9dbd4899-87r7z" event={"ID":"43d9ca79-ce89-424a-bb1b-dd8197cd5d6b","Type":"ContainerStarted","Data":"74344fc177ab50bc952fa901949e3b440f25cf9fa156faee80407bcace503ff6"}
Sep 14 10:06:47.134292 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:47.134232    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-51aefd/auth-test-model-51aefd-kserve-7c9dbd4899-87r7z" podStartSLOduration=1.134211696 podStartE2EDuration="1.134211696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:06:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:06:47.132282257 +0000 UTC m=+1396.171956878" watchObservedRunningTime="2026-09-14 10:06:47.134211696 +0000 UTC m=+1396.173886317"
Sep 14 10:06:49.141962 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:49.141928    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:06:49.241562 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:49.241522    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:06:49.309329 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:49.309282    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-75017ebc-q7s8n"]
Sep 14 10:06:49.312370 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:49.312330    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-75017ebc-q7s8n"]
Sep 14 10:06:49.313278 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:49.313232    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:06:49.370887 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:49.370840    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:06:49.522651 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:49.522591    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="817ca3cc-631b-40a4-bf2d-ce2e77dd3e24" path="/var/lib/kubelet/pods/817ca3cc-631b-40a4-bf2d-ce2e77dd3e24/volumes"
Sep 14 10:06:49.778473 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:49.778386    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:06:50.507845 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:50.507813    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:06:51.394034 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:51.393985    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:06:52.313985 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:52.313954    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:06:52.415606 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:52.415549    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:06:53.018202 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:53.018158    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:06:53.897739 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:53.897704    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:06:54.596731 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:54.596675    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:06:56.466262 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:56.466212    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-51aefd/auth-test-model-51aefd-kserve-7c9dbd4899-87r7z"
Sep 14 10:06:56.466685 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:56.466277    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-51aefd/auth-test-model-51aefd-kserve-7c9dbd4899-87r7z"
Sep 14 10:06:56.479279 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:56.479243    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-51aefd/auth-test-model-51aefd-kserve-7c9dbd4899-87r7z"
Sep 14 10:06:57.200085 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:57.200041    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-51aefd/auth-test-model-51aefd-kserve-7c9dbd4899-87r7z"
Sep 14 10:06:57.301566 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:57.301514    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-75017ebc/e2e-lifecycle-model-75017ebc-kserve-c88f6db99-29nzt"]
Sep 14 10:06:57.394958 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:06:57.394923    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:07:09.758859 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:09.758821    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:07:12.323551 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:12.323509    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-75017ebc/e2e-lifecycle-model-75017ebc-kserve-c88f6db99-29nzt" podUID="becc30c1-8ba3-4252-99d8-6a2c55441916" containerName="main" containerID="cri-o://f2a671c0b43416d95edce0330f96e7f9b4612de99f6f0b3b2e9035448b76c5ea" gracePeriod=45
Sep 14 10:07:12.590128 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:12.590051    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-75017ebc/e2e-lifecycle-model-75017ebc-kserve-c88f6db99-29nzt"
Sep 14 10:07:12.621561 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:12.621527    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/becc30c1-8ba3-4252-99d8-6a2c55441916-tls-certs\") pod \"becc30c1-8ba3-4252-99d8-6a2c55441916\" (UID: \"becc30c1-8ba3-4252-99d8-6a2c55441916\") "
Sep 14 10:07:12.621881 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:12.621860    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/becc30c1-8ba3-4252-99d8-6a2c55441916-tmp-dir\") pod \"becc30c1-8ba3-4252-99d8-6a2c55441916\" (UID: \"becc30c1-8ba3-4252-99d8-6a2c55441916\") "
Sep 14 10:07:12.622031 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:12.622015    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z5wjq\" (UniqueName: \"kubernetes.io/projected/becc30c1-8ba3-4252-99d8-6a2c55441916-kube-api-access-z5wjq\") pod \"becc30c1-8ba3-4252-99d8-6a2c55441916\" (UID: \"becc30c1-8ba3-4252-99d8-6a2c55441916\") "
Sep 14 10:07:12.622189 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:12.622176    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/becc30c1-8ba3-4252-99d8-6a2c55441916-home\") pod \"becc30c1-8ba3-4252-99d8-6a2c55441916\" (UID: \"becc30c1-8ba3-4252-99d8-6a2c55441916\") "
Sep 14 10:07:12.622295 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:12.622282    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/becc30c1-8ba3-4252-99d8-6a2c55441916-dshm\") pod \"becc30c1-8ba3-4252-99d8-6a2c55441916\" (UID: \"becc30c1-8ba3-4252-99d8-6a2c55441916\") "
Sep 14 10:07:12.622427 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:12.622415    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/becc30c1-8ba3-4252-99d8-6a2c55441916-model-cache\") pod \"becc30c1-8ba3-4252-99d8-6a2c55441916\" (UID: \"becc30c1-8ba3-4252-99d8-6a2c55441916\") "
Sep 14 10:07:12.623052 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:12.623025    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/becc30c1-8ba3-4252-99d8-6a2c55441916-model-cache" (OuterVolumeSpecName: "model-cache") pod "becc30c1-8ba3-4252-99d8-6a2c55441916" (UID: "becc30c1-8ba3-4252-99d8-6a2c55441916"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:07:12.623186 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:12.623126    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/becc30c1-8ba3-4252-99d8-6a2c55441916-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "becc30c1-8ba3-4252-99d8-6a2c55441916" (UID: "becc30c1-8ba3-4252-99d8-6a2c55441916"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:07:12.623328 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:12.623303    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/becc30c1-8ba3-4252-99d8-6a2c55441916-home" (OuterVolumeSpecName: "home") pod "becc30c1-8ba3-4252-99d8-6a2c55441916" (UID: "becc30c1-8ba3-4252-99d8-6a2c55441916"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:07:12.625363 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:12.625322    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/becc30c1-8ba3-4252-99d8-6a2c55441916-tmp-dir\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:07:12.625555 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:12.625521    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/becc30c1-8ba3-4252-99d8-6a2c55441916-home\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:07:12.625555 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:12.625548    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/becc30c1-8ba3-4252-99d8-6a2c55441916-model-cache\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:07:12.626168 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:12.626135    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/becc30c1-8ba3-4252-99d8-6a2c55441916-kube-api-access-z5wjq" (OuterVolumeSpecName: "kube-api-access-z5wjq") pod "becc30c1-8ba3-4252-99d8-6a2c55441916" (UID: "becc30c1-8ba3-4252-99d8-6a2c55441916"). InnerVolumeSpecName "kube-api-access-z5wjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:07:12.626306 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:12.626288    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/becc30c1-8ba3-4252-99d8-6a2c55441916-dshm" (OuterVolumeSpecName: "dshm") pod "becc30c1-8ba3-4252-99d8-6a2c55441916" (UID: "becc30c1-8ba3-4252-99d8-6a2c55441916"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:07:12.628711 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:12.628675    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/becc30c1-8ba3-4252-99d8-6a2c55441916-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "becc30c1-8ba3-4252-99d8-6a2c55441916" (UID: "becc30c1-8ba3-4252-99d8-6a2c55441916"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:07:12.726660 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:12.726615    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/becc30c1-8ba3-4252-99d8-6a2c55441916-dshm\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:07:12.726660 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:12.726654    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/becc30c1-8ba3-4252-99d8-6a2c55441916-tls-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:07:12.726660 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:12.726666    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5wjq\" (UniqueName: \"kubernetes.io/projected/becc30c1-8ba3-4252-99d8-6a2c55441916-kube-api-access-z5wjq\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:07:13.302309 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:13.302272    2563 generic.go:356] "Generic (PLEG): container finished" podID="becc30c1-8ba3-4252-99d8-6a2c55441916" containerID="f2a671c0b43416d95edce0330f96e7f9b4612de99f6f0b3b2e9035448b76c5ea" exitCode=0
Sep 14 10:07:13.302539 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:13.302333    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-75017ebc/e2e-lifecycle-model-75017ebc-kserve-c88f6db99-29nzt"
Sep 14 10:07:13.302539 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:13.302367    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-75017ebc/e2e-lifecycle-model-75017ebc-kserve-c88f6db99-29nzt" event={"ID":"becc30c1-8ba3-4252-99d8-6a2c55441916","Type":"ContainerDied","Data":"f2a671c0b43416d95edce0330f96e7f9b4612de99f6f0b3b2e9035448b76c5ea"}
Sep 14 10:07:13.302539 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:13.302424    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-75017ebc/e2e-lifecycle-model-75017ebc-kserve-c88f6db99-29nzt" event={"ID":"becc30c1-8ba3-4252-99d8-6a2c55441916","Type":"ContainerDied","Data":"1ed19d1f019245b4a6a5412b4d7606fe900b0938e6d65dd226c0c02035ca9ca7"}
Sep 14 10:07:13.302539 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:13.302448    2563 scope.go:117] "RemoveContainer" containerID="f2a671c0b43416d95edce0330f96e7f9b4612de99f6f0b3b2e9035448b76c5ea"
Sep 14 10:07:13.313300 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:13.313277    2563 scope.go:117] "RemoveContainer" containerID="f2a671c0b43416d95edce0330f96e7f9b4612de99f6f0b3b2e9035448b76c5ea"
Sep 14 10:07:13.313667 ip-10-0-132-4 kubenswrapper[2563]: E0914 10:07:13.313646    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2a671c0b43416d95edce0330f96e7f9b4612de99f6f0b3b2e9035448b76c5ea\": container with ID starting with f2a671c0b43416d95edce0330f96e7f9b4612de99f6f0b3b2e9035448b76c5ea not found: ID does not exist" containerID="f2a671c0b43416d95edce0330f96e7f9b4612de99f6f0b3b2e9035448b76c5ea"
Sep 14 10:07:13.313741 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:13.313678    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2a671c0b43416d95edce0330f96e7f9b4612de99f6f0b3b2e9035448b76c5ea"} err="failed to get container status \"f2a671c0b43416d95edce0330f96e7f9b4612de99f6f0b3b2e9035448b76c5ea\": rpc error: code = NotFound desc = could not find container \"f2a671c0b43416d95edce0330f96e7f9b4612de99f6f0b3b2e9035448b76c5ea\": container with ID starting with f2a671c0b43416d95edce0330f96e7f9b4612de99f6f0b3b2e9035448b76c5ea not found: ID does not exist"
Sep 14 10:07:13.325279 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:13.325237    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-75017ebc/e2e-lifecycle-model-75017ebc-kserve-c88f6db99-29nzt"]
Sep 14 10:07:13.328858 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:13.328822    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-75017ebc/e2e-lifecycle-model-75017ebc-kserve-c88f6db99-29nzt"]
Sep 14 10:07:13.483245 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:13.483210    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="becc30c1-8ba3-4252-99d8-6a2c55441916" path="/var/lib/kubelet/pods/becc30c1-8ba3-4252-99d8-6a2c55441916/volumes"
Sep 14 10:07:18.942593 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:18.942547    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:07:21.690482 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:21.690414    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:07:21.809281 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:21.809245    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:07:28.206255 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:28.206195    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:07:28.307207 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:28.307168    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:07:33.820829 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:33.820793    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:07:34.768061 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:34.767994    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:07:40.849670 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:40.849637    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:07:40.949523 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:40.949483    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:07:41.088911 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:41.088873    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5"]
Sep 14 10:07:41.089183 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:41.089158    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5" podUID="f7b2e299-2b37-4d21-9264-1cbf9d15c274" containerName="istio-proxy" containerID="cri-o://f93d12956a27cd5d8b73982ee0e944d15d2f7c9835d9e4fc25187c339cf5be1c" gracePeriod=30
Sep 14 10:07:41.113877 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:41.113779    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:07:42.431847 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:42.431799    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:07:42.580571 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:42.580528    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:07:43.153842 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:43.153799    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:07:43.254479 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:43.254442    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:07:43.354373 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:43.354332    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:07:44.652193 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:44.652155    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:07:46.353977 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:46.353902    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5"
Sep 14 10:07:46.398431 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:46.398394    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7b2e299-2b37-4d21-9264-1cbf9d15c274-workload-socket\") pod \"f7b2e299-2b37-4d21-9264-1cbf9d15c274\" (UID: \"f7b2e299-2b37-4d21-9264-1cbf9d15c274\") "
Sep 14 10:07:46.398633 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:46.398475    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7b2e299-2b37-4d21-9264-1cbf9d15c274-istio-envoy\") pod \"f7b2e299-2b37-4d21-9264-1cbf9d15c274\" (UID: \"f7b2e299-2b37-4d21-9264-1cbf9d15c274\") "
Sep 14 10:07:46.398633 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:46.398507    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7b2e299-2b37-4d21-9264-1cbf9d15c274-istio-data\") pod \"f7b2e299-2b37-4d21-9264-1cbf9d15c274\" (UID: \"f7b2e299-2b37-4d21-9264-1cbf9d15c274\") "
Sep 14 10:07:46.398633 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:46.398535    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7b2e299-2b37-4d21-9264-1cbf9d15c274-istio-podinfo\") pod \"f7b2e299-2b37-4d21-9264-1cbf9d15c274\" (UID: \"f7b2e299-2b37-4d21-9264-1cbf9d15c274\") "
Sep 14 10:07:46.398633 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:46.398572    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7b2e299-2b37-4d21-9264-1cbf9d15c274-istio-token\") pod \"f7b2e299-2b37-4d21-9264-1cbf9d15c274\" (UID: \"f7b2e299-2b37-4d21-9264-1cbf9d15c274\") "
Sep 14 10:07:46.398860 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:46.398649    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7b2e299-2b37-4d21-9264-1cbf9d15c274-workload-certs\") pod \"f7b2e299-2b37-4d21-9264-1cbf9d15c274\" (UID: \"f7b2e299-2b37-4d21-9264-1cbf9d15c274\") "
Sep 14 10:07:46.398860 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:46.398685    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7b2e299-2b37-4d21-9264-1cbf9d15c274-istiod-ca-cert\") pod \"f7b2e299-2b37-4d21-9264-1cbf9d15c274\" (UID: \"f7b2e299-2b37-4d21-9264-1cbf9d15c274\") "
Sep 14 10:07:46.398860 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:46.398751    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7b2e299-2b37-4d21-9264-1cbf9d15c274-credential-socket\") pod \"f7b2e299-2b37-4d21-9264-1cbf9d15c274\" (UID: \"f7b2e299-2b37-4d21-9264-1cbf9d15c274\") "
Sep 14 10:07:46.398860 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:46.398775    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7b2e299-2b37-4d21-9264-1cbf9d15c274-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f7b2e299-2b37-4d21-9264-1cbf9d15c274" (UID: "f7b2e299-2b37-4d21-9264-1cbf9d15c274"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:07:46.398860 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:46.398805    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cg87f\" (UniqueName: \"kubernetes.io/projected/f7b2e299-2b37-4d21-9264-1cbf9d15c274-kube-api-access-cg87f\") pod \"f7b2e299-2b37-4d21-9264-1cbf9d15c274\" (UID: \"f7b2e299-2b37-4d21-9264-1cbf9d15c274\") "
Sep 14 10:07:46.399173 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:46.398851    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7b2e299-2b37-4d21-9264-1cbf9d15c274-istio-data" (OuterVolumeSpecName: "istio-data") pod "f7b2e299-2b37-4d21-9264-1cbf9d15c274" (UID: "f7b2e299-2b37-4d21-9264-1cbf9d15c274"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:07:46.399173 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:46.399146    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7b2e299-2b37-4d21-9264-1cbf9d15c274-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f7b2e299-2b37-4d21-9264-1cbf9d15c274" (UID: "f7b2e299-2b37-4d21-9264-1cbf9d15c274"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:07:46.399296 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:46.399230    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7b2e299-2b37-4d21-9264-1cbf9d15c274-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f7b2e299-2b37-4d21-9264-1cbf9d15c274" (UID: "f7b2e299-2b37-4d21-9264-1cbf9d15c274"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:07:46.399296 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:46.399249    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7b2e299-2b37-4d21-9264-1cbf9d15c274-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f7b2e299-2b37-4d21-9264-1cbf9d15c274" (UID: "f7b2e299-2b37-4d21-9264-1cbf9d15c274"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:07:46.399438 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:46.399413    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7b2e299-2b37-4d21-9264-1cbf9d15c274-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:07:46.399438 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:46.399440    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7b2e299-2b37-4d21-9264-1cbf9d15c274-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:07:46.399644 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:46.399455    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7b2e299-2b37-4d21-9264-1cbf9d15c274-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:07:46.399644 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:46.399468    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7b2e299-2b37-4d21-9264-1cbf9d15c274-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:07:46.399644 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:46.399480    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7b2e299-2b37-4d21-9264-1cbf9d15c274-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:07:46.401227 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:46.401193    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7b2e299-2b37-4d21-9264-1cbf9d15c274-kube-api-access-cg87f" (OuterVolumeSpecName: "kube-api-access-cg87f") pod "f7b2e299-2b37-4d21-9264-1cbf9d15c274" (UID: "f7b2e299-2b37-4d21-9264-1cbf9d15c274"). InnerVolumeSpecName "kube-api-access-cg87f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:07:46.401521 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:46.401481    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7b2e299-2b37-4d21-9264-1cbf9d15c274-istio-token" (OuterVolumeSpecName: "istio-token") pod "f7b2e299-2b37-4d21-9264-1cbf9d15c274" (UID: "f7b2e299-2b37-4d21-9264-1cbf9d15c274"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:07:46.401709 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:46.401685    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7b2e299-2b37-4d21-9264-1cbf9d15c274-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f7b2e299-2b37-4d21-9264-1cbf9d15c274" (UID: "f7b2e299-2b37-4d21-9264-1cbf9d15c274"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:07:46.401790 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:46.401750    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f7b2e299-2b37-4d21-9264-1cbf9d15c274-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f7b2e299-2b37-4d21-9264-1cbf9d15c274" (UID: "f7b2e299-2b37-4d21-9264-1cbf9d15c274"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 10:07:46.500975 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:46.500933    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cg87f\" (UniqueName: \"kubernetes.io/projected/f7b2e299-2b37-4d21-9264-1cbf9d15c274-kube-api-access-cg87f\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:07:46.500975 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:46.500969    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7b2e299-2b37-4d21-9264-1cbf9d15c274-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:07:46.500975 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:46.500979    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7b2e299-2b37-4d21-9264-1cbf9d15c274-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:07:46.501239 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:46.500989    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7b2e299-2b37-4d21-9264-1cbf9d15c274-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:07:46.531725 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:46.531691    2563 generic.go:356] "Generic (PLEG): container finished" podID="f7b2e299-2b37-4d21-9264-1cbf9d15c274" containerID="f93d12956a27cd5d8b73982ee0e944d15d2f7c9835d9e4fc25187c339cf5be1c" exitCode=0
Sep 14 10:07:46.531888 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:46.531746    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5"
Sep 14 10:07:46.531888 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:46.531780    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5" event={"ID":"f7b2e299-2b37-4d21-9264-1cbf9d15c274","Type":"ContainerDied","Data":"f93d12956a27cd5d8b73982ee0e944d15d2f7c9835d9e4fc25187c339cf5be1c"}
Sep 14 10:07:46.531888 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:46.531841    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5" event={"ID":"f7b2e299-2b37-4d21-9264-1cbf9d15c274","Type":"ContainerDied","Data":"69b877abb2b543a923347c2669409407fe8f0668a785147c52e9be7819747a30"}
Sep 14 10:07:46.532468 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:46.531867    2563 scope.go:117] "RemoveContainer" containerID="f93d12956a27cd5d8b73982ee0e944d15d2f7c9835d9e4fc25187c339cf5be1c"
Sep 14 10:07:46.543168 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:46.543147    2563 scope.go:117] "RemoveContainer" containerID="bd56ee38bd9e54dada49c06a4a0c2cc9a307672c3314a785c10e1a1ad74e855b"
Sep 14 10:07:46.554805 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:46.554776    2563 scope.go:117] "RemoveContainer" containerID="f93d12956a27cd5d8b73982ee0e944d15d2f7c9835d9e4fc25187c339cf5be1c"
Sep 14 10:07:46.555226 ip-10-0-132-4 kubenswrapper[2563]: E0914 10:07:46.555201    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f93d12956a27cd5d8b73982ee0e944d15d2f7c9835d9e4fc25187c339cf5be1c\": container with ID starting with f93d12956a27cd5d8b73982ee0e944d15d2f7c9835d9e4fc25187c339cf5be1c not found: ID does not exist" containerID="f93d12956a27cd5d8b73982ee0e944d15d2f7c9835d9e4fc25187c339cf5be1c"
Sep 14 10:07:46.555338 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:46.555240    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f93d12956a27cd5d8b73982ee0e944d15d2f7c9835d9e4fc25187c339cf5be1c"} err="failed to get container status \"f93d12956a27cd5d8b73982ee0e944d15d2f7c9835d9e4fc25187c339cf5be1c\": rpc error: code = NotFound desc = could not find container \"f93d12956a27cd5d8b73982ee0e944d15d2f7c9835d9e4fc25187c339cf5be1c\": container with ID starting with f93d12956a27cd5d8b73982ee0e944d15d2f7c9835d9e4fc25187c339cf5be1c not found: ID does not exist"
Sep 14 10:07:46.555338 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:46.555267    2563 scope.go:117] "RemoveContainer" containerID="bd56ee38bd9e54dada49c06a4a0c2cc9a307672c3314a785c10e1a1ad74e855b"
Sep 14 10:07:46.555552 ip-10-0-132-4 kubenswrapper[2563]: E0914 10:07:46.555533    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd56ee38bd9e54dada49c06a4a0c2cc9a307672c3314a785c10e1a1ad74e855b\": container with ID starting with bd56ee38bd9e54dada49c06a4a0c2cc9a307672c3314a785c10e1a1ad74e855b not found: ID does not exist" containerID="bd56ee38bd9e54dada49c06a4a0c2cc9a307672c3314a785c10e1a1ad74e855b"
Sep 14 10:07:46.555615 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:46.555560    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd56ee38bd9e54dada49c06a4a0c2cc9a307672c3314a785c10e1a1ad74e855b"} err="failed to get container status \"bd56ee38bd9e54dada49c06a4a0c2cc9a307672c3314a785c10e1a1ad74e855b\": rpc error: code = NotFound desc = could not find container \"bd56ee38bd9e54dada49c06a4a0c2cc9a307672c3314a785c10e1a1ad74e855b\": container with ID starting with bd56ee38bd9e54dada49c06a4a0c2cc9a307672c3314a785c10e1a1ad74e855b not found: ID does not exist"
Sep 14 10:07:46.557912 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:46.557879    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5"]
Sep 14 10:07:46.562817 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:46.562777    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-75017ebc-openshift-default-5c6448758d-sd4b5"]
Sep 14 10:07:46.930624 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:46.930593    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:07:47.483327 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:47.483290    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7b2e299-2b37-4d21-9264-1cbf9d15c274" path="/var/lib/kubelet/pods/f7b2e299-2b37-4d21-9264-1cbf9d15c274/volumes"
Sep 14 10:07:50.874735 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:50.874696    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:07:51.105281 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:51.105245    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:07:51.247719 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:51.247681    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:07:51.867850 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:51.867812    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:07:53.673301 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:53.673265    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:07:58.142691 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:58.142656    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:07:58.209615 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:58.209579    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:07:58.320245 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:58.320203    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:07:58.455790 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:07:58.455700    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:08:08.833923 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:08:08.833880    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:08:09.057785 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:08:09.057749    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:08:13.856655 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:08:13.856607    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-f4d054/auth-test-model-f4d054-kserve-b4655dc8c-4zclc"]
Sep 14 10:08:13.857097 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:08:13.857076    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="becc30c1-8ba3-4252-99d8-6a2c55441916" containerName="main"
Sep 14 10:08:13.857097 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:08:13.857094    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="becc30c1-8ba3-4252-99d8-6a2c55441916" containerName="main"
Sep 14 10:08:13.857180 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:08:13.857116    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7b2e299-2b37-4d21-9264-1cbf9d15c274" containerName="istio-proxy"
Sep 14 10:08:13.857180 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:08:13.857122    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7b2e299-2b37-4d21-9264-1cbf9d15c274" containerName="istio-proxy"
Sep 14 10:08:13.857180 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:08:13.857147    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7b2e299-2b37-4d21-9264-1cbf9d15c274" containerName="istio-proxy"
Sep 14 10:08:13.857180 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:08:13.857155    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7b2e299-2b37-4d21-9264-1cbf9d15c274" containerName="istio-proxy"
Sep 14 10:08:13.857307 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:08:13.857223    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7b2e299-2b37-4d21-9264-1cbf9d15c274" containerName="istio-proxy"
Sep 14 10:08:13.857307 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:08:13.857231    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="becc30c1-8ba3-4252-99d8-6a2c55441916" containerName="main"
Sep 14 10:08:13.860485 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:08:13.860462    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f4d054/auth-test-model-f4d054-kserve-b4655dc8c-4zclc"
Sep 14 10:08:13.862775 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:08:13.862748    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-f4d054\"/\"openshift-service-ca.crt\""
Sep 14 10:08:13.863401 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:08:13.863379    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-f4d054\"/\"kube-root-ca.crt\""
Sep 14 10:08:13.863529 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:08:13.863420    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-f4d054\"/\"auth-test-model-f4d054-kserve-self-signed-certs\""
Sep 14 10:08:13.863529 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:08:13.863456    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-f4d054\"/\"default-dockercfg-wnn54\""
Sep 14 10:08:13.874767 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:08:13.874731    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f4d054/auth-test-model-f4d054-kserve-b4655dc8c-4zclc"]
Sep 14 10:08:13.910547 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:08:13.910505    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4db918fc-87ba-418e-b581-cad6bed75247-model-cache\") pod \"auth-test-model-f4d054-kserve-b4655dc8c-4zclc\" (UID: \"4db918fc-87ba-418e-b581-cad6bed75247\") " pod="ai-tenant-e2e-shared-b-w3-f4d054/auth-test-model-f4d054-kserve-b4655dc8c-4zclc"
Sep 14 10:08:13.910756 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:08:13.910630    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4db918fc-87ba-418e-b581-cad6bed75247-home\") pod \"auth-test-model-f4d054-kserve-b4655dc8c-4zclc\" (UID: \"4db918fc-87ba-418e-b581-cad6bed75247\") " pod="ai-tenant-e2e-shared-b-w3-f4d054/auth-test-model-f4d054-kserve-b4655dc8c-4zclc"
Sep 14 10:08:13.910756 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:08:13.910688    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4db918fc-87ba-418e-b581-cad6bed75247-dshm\") pod \"auth-test-model-f4d054-kserve-b4655dc8c-4zclc\" (UID: \"4db918fc-87ba-418e-b581-cad6bed75247\") " pod="ai-tenant-e2e-shared-b-w3-f4d054/auth-test-model-f4d054-kserve-b4655dc8c-4zclc"
Sep 14 10:08:13.910907 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:08:13.910782    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4db918fc-87ba-418e-b581-cad6bed75247-tls-certs\") pod \"auth-test-model-f4d054-kserve-b4655dc8c-4zclc\" (UID: \"4db918fc-87ba-418e-b581-cad6bed75247\") " pod="ai-tenant-e2e-shared-b-w3-f4d054/auth-test-model-f4d054-kserve-b4655dc8c-4zclc"
Sep 14 10:08:13.910907 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:08:13.910857    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjvml\" (UniqueName: \"kubernetes.io/projected/4db918fc-87ba-418e-b581-cad6bed75247-kube-api-access-hjvml\") pod \"auth-test-model-f4d054-kserve-b4655dc8c-4zclc\" (UID: \"4db918fc-87ba-418e-b581-cad6bed75247\") " pod="ai-tenant-e2e-shared-b-w3-f4d054/auth-test-model-f4d054-kserve-b4655dc8c-4zclc"
Sep 14 10:08:13.911098 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:08:13.910909    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4db918fc-87ba-418e-b581-cad6bed75247-tmp-dir\") pod \"auth-test-model-f4d054-kserve-b4655dc8c-4zclc\" (UID: \"4db918fc-87ba-418e-b581-cad6bed75247\") " pod="ai-tenant-e2e-shared-b-w3-f4d054/auth-test-model-f4d054-kserve-b4655dc8c-4zclc"
Sep 14 10:08:13.942689 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:08:13.942650    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:08:14.012123 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:08:14.012088    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4db918fc-87ba-418e-b581-cad6bed75247-model-cache\") pod \"auth-test-model-f4d054-kserve-b4655dc8c-4zclc\" (UID: \"4db918fc-87ba-418e-b581-cad6bed75247\") " pod="ai-tenant-e2e-shared-b-w3-f4d054/auth-test-model-f4d054-kserve-b4655dc8c-4zclc"
Sep 14 10:08:14.012292 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:08:14.012168    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4db918fc-87ba-418e-b581-cad6bed75247-home\") pod \"auth-test-model-f4d054-kserve-b4655dc8c-4zclc\" (UID: \"4db918fc-87ba-418e-b581-cad6bed75247\") " pod="ai-tenant-e2e-shared-b-w3-f4d054/auth-test-model-f4d054-kserve-b4655dc8c-4zclc"
Sep 14 10:08:14.012292 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:08:14.012190    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4db918fc-87ba-418e-b581-cad6bed75247-dshm\") pod \"auth-test-model-f4d054-kserve-b4655dc8c-4zclc\" (UID: \"4db918fc-87ba-418e-b581-cad6bed75247\") " pod="ai-tenant-e2e-shared-b-w3-f4d054/auth-test-model-f4d054-kserve-b4655dc8c-4zclc"
Sep 14 10:08:14.012292 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:08:14.012229    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4db918fc-87ba-418e-b581-cad6bed75247-tls-certs\") pod \"auth-test-model-f4d054-kserve-b4655dc8c-4zclc\" (UID: \"4db918fc-87ba-418e-b581-cad6bed75247\") " pod="ai-tenant-e2e-shared-b-w3-f4d054/auth-test-model-f4d054-kserve-b4655dc8c-4zclc"
Sep 14 10:08:14.012292 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:08:14.012267    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjvml\" (UniqueName: \"kubernetes.io/projected/4db918fc-87ba-418e-b581-cad6bed75247-kube-api-access-hjvml\") pod \"auth-test-model-f4d054-kserve-b4655dc8c-4zclc\" (UID: \"4db918fc-87ba-418e-b581-cad6bed75247\") " pod="ai-tenant-e2e-shared-b-w3-f4d054/auth-test-model-f4d054-kserve-b4655dc8c-4zclc"
Sep 14 10:08:14.012484 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:08:14.012294    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4db918fc-87ba-418e-b581-cad6bed75247-tmp-dir\") pod \"auth-test-model-f4d054-kserve-b4655dc8c-4zclc\" (UID: \"4db918fc-87ba-418e-b581-cad6bed75247\") " pod="ai-tenant-e2e-shared-b-w3-f4d054/auth-test-model-f4d054-kserve-b4655dc8c-4zclc"
Sep 14 10:08:14.012618 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:08:14.012589    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4db918fc-87ba-418e-b581-cad6bed75247-model-cache\") pod \"auth-test-model-f4d054-kserve-b4655dc8c-4zclc\" (UID: \"4db918fc-87ba-418e-b581-cad6bed75247\") " pod="ai-tenant-e2e-shared-b-w3-f4d054/auth-test-model-f4d054-kserve-b4655dc8c-4zclc"
Sep 14 10:08:14.012689 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:08:14.012628    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4db918fc-87ba-418e-b581-cad6bed75247-home\") pod \"auth-test-model-f4d054-kserve-b4655dc8c-4zclc\" (UID: \"4db918fc-87ba-418e-b581-cad6bed75247\") " pod="ai-tenant-e2e-shared-b-w3-f4d054/auth-test-model-f4d054-kserve-b4655dc8c-4zclc"
Sep 14 10:08:14.012732 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:08:14.012684    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4db918fc-87ba-418e-b581-cad6bed75247-tmp-dir\") pod \"auth-test-model-f4d054-kserve-b4655dc8c-4zclc\" (UID: \"4db918fc-87ba-418e-b581-cad6bed75247\") " pod="ai-tenant-e2e-shared-b-w3-f4d054/auth-test-model-f4d054-kserve-b4655dc8c-4zclc"
Sep 14 10:08:14.014847 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:08:14.014821    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4db918fc-87ba-418e-b581-cad6bed75247-dshm\") pod \"auth-test-model-f4d054-kserve-b4655dc8c-4zclc\" (UID: \"4db918fc-87ba-418e-b581-cad6bed75247\") " pod="ai-tenant-e2e-shared-b-w3-f4d054/auth-test-model-f4d054-kserve-b4655dc8c-4zclc"
Sep 14 10:08:14.014985 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:08:14.014854    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4db918fc-87ba-418e-b581-cad6bed75247-tls-certs\") pod \"auth-test-model-f4d054-kserve-b4655dc8c-4zclc\" (UID: \"4db918fc-87ba-418e-b581-cad6bed75247\") " pod="ai-tenant-e2e-shared-b-w3-f4d054/auth-test-model-f4d054-kserve-b4655dc8c-4zclc"
Sep 14 10:08:14.023190 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:08:14.023144    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjvml\" (UniqueName: \"kubernetes.io/projected/4db918fc-87ba-418e-b581-cad6bed75247-kube-api-access-hjvml\") pod \"auth-test-model-f4d054-kserve-b4655dc8c-4zclc\" (UID: \"4db918fc-87ba-418e-b581-cad6bed75247\") " pod="ai-tenant-e2e-shared-b-w3-f4d054/auth-test-model-f4d054-kserve-b4655dc8c-4zclc"
Sep 14 10:08:14.171551 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:08:14.171449    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f4d054/auth-test-model-f4d054-kserve-b4655dc8c-4zclc"
Sep 14 10:08:14.457091 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:08:14.457053    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f4d054/auth-test-model-f4d054-kserve-b4655dc8c-4zclc"]
Sep 14 10:08:14.458413 ip-10-0-132-4 kubenswrapper[2563]: W0914 10:08:14.458383    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4db918fc_87ba_418e_b581_cad6bed75247.slice/crio-712f17f87ea5035cb57f0b09ace9c01ac880b2b026dc809252c4a86aaab9badd WatchSource:0}: Error finding container 712f17f87ea5035cb57f0b09ace9c01ac880b2b026dc809252c4a86aaab9badd: Status 404 returned error can't find the container with id 712f17f87ea5035cb57f0b09ace9c01ac880b2b026dc809252c4a86aaab9badd
Sep 14 10:08:14.732678 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:08:14.732576    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f4d054/auth-test-model-f4d054-kserve-b4655dc8c-4zclc" event={"ID":"4db918fc-87ba-418e-b581-cad6bed75247","Type":"ContainerStarted","Data":"ecb16bd10fe1e9577ada85361634960a380c22d1ee50960227eeb3414ab8025d"}
Sep 14 10:08:14.732678 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:08:14.732619    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f4d054/auth-test-model-f4d054-kserve-b4655dc8c-4zclc" event={"ID":"4db918fc-87ba-418e-b581-cad6bed75247","Type":"ContainerStarted","Data":"712f17f87ea5035cb57f0b09ace9c01ac880b2b026dc809252c4a86aaab9badd"}
Sep 14 10:08:14.751841 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:08:14.751778    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-f4d054/auth-test-model-f4d054-kserve-b4655dc8c-4zclc" podStartSLOduration=1.751759283 podStartE2EDuration="1.751759283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:08:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:08:14.750060486 +0000 UTC m=+1483.789735107" watchObservedRunningTime="2026-09-14 10:08:14.751759283 +0000 UTC m=+1483.791433904"
Sep 14 10:08:24.172423 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:08:24.172364    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-f4d054/auth-test-model-f4d054-kserve-b4655dc8c-4zclc"
Sep 14 10:08:24.172423 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:08:24.172435    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-f4d054/auth-test-model-f4d054-kserve-b4655dc8c-4zclc"
Sep 14 10:08:24.185344 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:08:24.185309    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-f4d054/auth-test-model-f4d054-kserve-b4655dc8c-4zclc"
Sep 14 10:08:24.836409 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:08:24.836376    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-f4d054/auth-test-model-f4d054-kserve-b4655dc8c-4zclc"
Sep 14 10:08:25.203908 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:08:25.203822    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:08:29.945505 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:08:29.945466    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:08:30.456515 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:08:30.456470    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:08:31.786242 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:08:31.786211    2563 scope.go:117] "RemoveContainer" containerID="7b4e67cb3ca24e3bb1716a6ff04607c4c2fd3f61c4ae13533758c511e0373103"
Sep 14 10:08:39.499674 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:08:39.499624    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:08:39.687232 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:08:39.687195    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:08:41.914397 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:08:41.914361    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:08:42.453068 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:08:42.453026    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:08:42.550982 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:08:42.550950    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:08:43.101797 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:08:43.101759    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:08:43.923236 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:08:43.923199    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:08:44.622261 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:08:44.622223    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:08:45.548364 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:08:45.548324    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:08:46.569577 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:08:46.569535    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:08:47.450444 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:08:47.450396    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:08:48.295304 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:08:48.295270    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:08:48.411823 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:08:48.411776    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:08:48.512199 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:08:48.512154    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:08:48.809453 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:08:48.809420    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:08:48.917076 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:08:48.917039    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:08:53.882678 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:08:53.882582    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:09:03.311349 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:03.311310    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:09:03.370320 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:03.370288    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:09:09.425267 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:09.425227    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:09:11.117300 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:11.117261    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:09:11.922251 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:11.922203    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:09:12.423316 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:12.423278    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:09:12.884394 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:12.884355    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:09:12.984720 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:12.984686    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:09:13.156959 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:13.156870    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-03fd6d-openshift-default-5854564496-x76m9"]
Sep 14 10:09:13.157434 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:13.157418    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7b2e299-2b37-4d21-9264-1cbf9d15c274" containerName="istio-proxy"
Sep 14 10:09:13.160789 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:13.160762    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-03fd6d-openshift-default-5854564496-x76m9"
Sep 14 10:09:13.163144 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:13.163116    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-03fd6d-openshift-default-dockercfg-zvrlh\""
Sep 14 10:09:13.174897 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:13.174858    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-03fd6d-openshift-default-5854564496-x76m9"]
Sep 14 10:09:13.187832 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:13.187796    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:09:13.275229 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:13.275171    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljb4j\" (UniqueName: \"kubernetes.io/projected/3650d5df-8b7e-465f-83c2-65d3daecd3de-kube-api-access-ljb4j\") pod \"e2e-api-b-03fd6d-openshift-default-5854564496-x76m9\" (UID: \"3650d5df-8b7e-465f-83c2-65d3daecd3de\") " pod="openshift-ingress/e2e-api-b-03fd6d-openshift-default-5854564496-x76m9"
Sep 14 10:09:13.275445 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:13.275294    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3650d5df-8b7e-465f-83c2-65d3daecd3de-credential-socket\") pod \"e2e-api-b-03fd6d-openshift-default-5854564496-x76m9\" (UID: \"3650d5df-8b7e-465f-83c2-65d3daecd3de\") " pod="openshift-ingress/e2e-api-b-03fd6d-openshift-default-5854564496-x76m9"
Sep 14 10:09:13.275445 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:13.275329    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3650d5df-8b7e-465f-83c2-65d3daecd3de-istio-envoy\") pod \"e2e-api-b-03fd6d-openshift-default-5854564496-x76m9\" (UID: \"3650d5df-8b7e-465f-83c2-65d3daecd3de\") " pod="openshift-ingress/e2e-api-b-03fd6d-openshift-default-5854564496-x76m9"
Sep 14 10:09:13.275445 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:13.275355    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3650d5df-8b7e-465f-83c2-65d3daecd3de-istio-token\") pod \"e2e-api-b-03fd6d-openshift-default-5854564496-x76m9\" (UID: \"3650d5df-8b7e-465f-83c2-65d3daecd3de\") " pod="openshift-ingress/e2e-api-b-03fd6d-openshift-default-5854564496-x76m9"
Sep 14 10:09:13.275445 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:13.275384    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3650d5df-8b7e-465f-83c2-65d3daecd3de-workload-socket\") pod \"e2e-api-b-03fd6d-openshift-default-5854564496-x76m9\" (UID: \"3650d5df-8b7e-465f-83c2-65d3daecd3de\") " pod="openshift-ingress/e2e-api-b-03fd6d-openshift-default-5854564496-x76m9"
Sep 14 10:09:13.275445 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:13.275406    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3650d5df-8b7e-465f-83c2-65d3daecd3de-istio-data\") pod \"e2e-api-b-03fd6d-openshift-default-5854564496-x76m9\" (UID: \"3650d5df-8b7e-465f-83c2-65d3daecd3de\") " pod="openshift-ingress/e2e-api-b-03fd6d-openshift-default-5854564496-x76m9"
Sep 14 10:09:13.275689 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:13.275520    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3650d5df-8b7e-465f-83c2-65d3daecd3de-istio-podinfo\") pod \"e2e-api-b-03fd6d-openshift-default-5854564496-x76m9\" (UID: \"3650d5df-8b7e-465f-83c2-65d3daecd3de\") " pod="openshift-ingress/e2e-api-b-03fd6d-openshift-default-5854564496-x76m9"
Sep 14 10:09:13.275689 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:13.275551    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3650d5df-8b7e-465f-83c2-65d3daecd3de-workload-certs\") pod \"e2e-api-b-03fd6d-openshift-default-5854564496-x76m9\" (UID: \"3650d5df-8b7e-465f-83c2-65d3daecd3de\") " pod="openshift-ingress/e2e-api-b-03fd6d-openshift-default-5854564496-x76m9"
Sep 14 10:09:13.275689 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:13.275595    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3650d5df-8b7e-465f-83c2-65d3daecd3de-istiod-ca-cert\") pod \"e2e-api-b-03fd6d-openshift-default-5854564496-x76m9\" (UID: \"3650d5df-8b7e-465f-83c2-65d3daecd3de\") " pod="openshift-ingress/e2e-api-b-03fd6d-openshift-default-5854564496-x76m9"
Sep 14 10:09:13.376804 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:13.376765    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3650d5df-8b7e-465f-83c2-65d3daecd3de-istio-podinfo\") pod \"e2e-api-b-03fd6d-openshift-default-5854564496-x76m9\" (UID: \"3650d5df-8b7e-465f-83c2-65d3daecd3de\") " pod="openshift-ingress/e2e-api-b-03fd6d-openshift-default-5854564496-x76m9"
Sep 14 10:09:13.376804 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:13.376808    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3650d5df-8b7e-465f-83c2-65d3daecd3de-workload-certs\") pod \"e2e-api-b-03fd6d-openshift-default-5854564496-x76m9\" (UID: \"3650d5df-8b7e-465f-83c2-65d3daecd3de\") " pod="openshift-ingress/e2e-api-b-03fd6d-openshift-default-5854564496-x76m9"
Sep 14 10:09:13.377079 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:13.376935    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3650d5df-8b7e-465f-83c2-65d3daecd3de-istiod-ca-cert\") pod \"e2e-api-b-03fd6d-openshift-default-5854564496-x76m9\" (UID: \"3650d5df-8b7e-465f-83c2-65d3daecd3de\") " pod="openshift-ingress/e2e-api-b-03fd6d-openshift-default-5854564496-x76m9"
Sep 14 10:09:13.377079 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:13.377021    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljb4j\" (UniqueName: \"kubernetes.io/projected/3650d5df-8b7e-465f-83c2-65d3daecd3de-kube-api-access-ljb4j\") pod \"e2e-api-b-03fd6d-openshift-default-5854564496-x76m9\" (UID: \"3650d5df-8b7e-465f-83c2-65d3daecd3de\") " pod="openshift-ingress/e2e-api-b-03fd6d-openshift-default-5854564496-x76m9"
Sep 14 10:09:13.377204 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:13.377126    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3650d5df-8b7e-465f-83c2-65d3daecd3de-credential-socket\") pod \"e2e-api-b-03fd6d-openshift-default-5854564496-x76m9\" (UID: \"3650d5df-8b7e-465f-83c2-65d3daecd3de\") " pod="openshift-ingress/e2e-api-b-03fd6d-openshift-default-5854564496-x76m9"
Sep 14 10:09:13.377204 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:13.377159    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3650d5df-8b7e-465f-83c2-65d3daecd3de-istio-envoy\") pod \"e2e-api-b-03fd6d-openshift-default-5854564496-x76m9\" (UID: \"3650d5df-8b7e-465f-83c2-65d3daecd3de\") " pod="openshift-ingress/e2e-api-b-03fd6d-openshift-default-5854564496-x76m9"
Sep 14 10:09:13.377204 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:13.377186    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3650d5df-8b7e-465f-83c2-65d3daecd3de-istio-token\") pod \"e2e-api-b-03fd6d-openshift-default-5854564496-x76m9\" (UID: \"3650d5df-8b7e-465f-83c2-65d3daecd3de\") " pod="openshift-ingress/e2e-api-b-03fd6d-openshift-default-5854564496-x76m9"
Sep 14 10:09:13.377375 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:13.377217    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3650d5df-8b7e-465f-83c2-65d3daecd3de-workload-socket\") pod \"e2e-api-b-03fd6d-openshift-default-5854564496-x76m9\" (UID: \"3650d5df-8b7e-465f-83c2-65d3daecd3de\") " pod="openshift-ingress/e2e-api-b-03fd6d-openshift-default-5854564496-x76m9"
Sep 14 10:09:13.377375 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:13.377216    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3650d5df-8b7e-465f-83c2-65d3daecd3de-workload-certs\") pod \"e2e-api-b-03fd6d-openshift-default-5854564496-x76m9\" (UID: \"3650d5df-8b7e-465f-83c2-65d3daecd3de\") " pod="openshift-ingress/e2e-api-b-03fd6d-openshift-default-5854564496-x76m9"
Sep 14 10:09:13.377375 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:13.377319    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3650d5df-8b7e-465f-83c2-65d3daecd3de-istio-data\") pod \"e2e-api-b-03fd6d-openshift-default-5854564496-x76m9\" (UID: \"3650d5df-8b7e-465f-83c2-65d3daecd3de\") " pod="openshift-ingress/e2e-api-b-03fd6d-openshift-default-5854564496-x76m9"
Sep 14 10:09:13.377614 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:13.377588    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3650d5df-8b7e-465f-83c2-65d3daecd3de-credential-socket\") pod \"e2e-api-b-03fd6d-openshift-default-5854564496-x76m9\" (UID: \"3650d5df-8b7e-465f-83c2-65d3daecd3de\") " pod="openshift-ingress/e2e-api-b-03fd6d-openshift-default-5854564496-x76m9"
Sep 14 10:09:13.377688 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:13.377623    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3650d5df-8b7e-465f-83c2-65d3daecd3de-workload-socket\") pod \"e2e-api-b-03fd6d-openshift-default-5854564496-x76m9\" (UID: \"3650d5df-8b7e-465f-83c2-65d3daecd3de\") " pod="openshift-ingress/e2e-api-b-03fd6d-openshift-default-5854564496-x76m9"
Sep 14 10:09:13.377750 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:13.377708    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3650d5df-8b7e-465f-83c2-65d3daecd3de-istiod-ca-cert\") pod \"e2e-api-b-03fd6d-openshift-default-5854564496-x76m9\" (UID: \"3650d5df-8b7e-465f-83c2-65d3daecd3de\") " pod="openshift-ingress/e2e-api-b-03fd6d-openshift-default-5854564496-x76m9"
Sep 14 10:09:13.377750 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:13.377725    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3650d5df-8b7e-465f-83c2-65d3daecd3de-istio-data\") pod \"e2e-api-b-03fd6d-openshift-default-5854564496-x76m9\" (UID: \"3650d5df-8b7e-465f-83c2-65d3daecd3de\") " pod="openshift-ingress/e2e-api-b-03fd6d-openshift-default-5854564496-x76m9"
Sep 14 10:09:13.379507 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:13.379473    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3650d5df-8b7e-465f-83c2-65d3daecd3de-istio-envoy\") pod \"e2e-api-b-03fd6d-openshift-default-5854564496-x76m9\" (UID: \"3650d5df-8b7e-465f-83c2-65d3daecd3de\") " pod="openshift-ingress/e2e-api-b-03fd6d-openshift-default-5854564496-x76m9"
Sep 14 10:09:13.379636 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:13.379549    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3650d5df-8b7e-465f-83c2-65d3daecd3de-istio-podinfo\") pod \"e2e-api-b-03fd6d-openshift-default-5854564496-x76m9\" (UID: \"3650d5df-8b7e-465f-83c2-65d3daecd3de\") " pod="openshift-ingress/e2e-api-b-03fd6d-openshift-default-5854564496-x76m9"
Sep 14 10:09:13.385669 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:13.385581    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3650d5df-8b7e-465f-83c2-65d3daecd3de-istio-token\") pod \"e2e-api-b-03fd6d-openshift-default-5854564496-x76m9\" (UID: \"3650d5df-8b7e-465f-83c2-65d3daecd3de\") " pod="openshift-ingress/e2e-api-b-03fd6d-openshift-default-5854564496-x76m9"
Sep 14 10:09:13.385840 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:13.385719    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljb4j\" (UniqueName: \"kubernetes.io/projected/3650d5df-8b7e-465f-83c2-65d3daecd3de-kube-api-access-ljb4j\") pod \"e2e-api-b-03fd6d-openshift-default-5854564496-x76m9\" (UID: \"3650d5df-8b7e-465f-83c2-65d3daecd3de\") " pod="openshift-ingress/e2e-api-b-03fd6d-openshift-default-5854564496-x76m9"
Sep 14 10:09:13.473168 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:13.473063    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-03fd6d-openshift-default-5854564496-x76m9"
Sep 14 10:09:13.712623 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:13.712571    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:09:13.760091 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:13.760046    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-03fd6d-openshift-default-5854564496-x76m9"]
Sep 14 10:09:13.761187 ip-10-0-132-4 kubenswrapper[2563]: W0914 10:09:13.761152    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3650d5df_8b7e_465f_83c2_65d3daecd3de.slice/crio-aecd8f4d2169f0d08391956e528c1d3bf7889f1455066496ede8a9aac32e67b2 WatchSource:0}: Error finding container aecd8f4d2169f0d08391956e528c1d3bf7889f1455066496ede8a9aac32e67b2: Status 404 returned error can't find the container with id aecd8f4d2169f0d08391956e528c1d3bf7889f1455066496ede8a9aac32e67b2
Sep 14 10:09:13.763775 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:13.763739    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 10:09:13.763885 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:13.763814    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 10:09:13.763885 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:13.763846    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 10:09:13.813083 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:13.813045    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:09:14.165510 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:14.165408    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-03fd6d-openshift-default-5854564496-x76m9" event={"ID":"3650d5df-8b7e-465f-83c2-65d3daecd3de","Type":"ContainerStarted","Data":"0d4a420373f7d148687d32fe245e7f2b7076632a6592f2ce61a42e9dd8643767"}
Sep 14 10:09:14.165510 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:14.165475    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-03fd6d-openshift-default-5854564496-x76m9" event={"ID":"3650d5df-8b7e-465f-83c2-65d3daecd3de","Type":"ContainerStarted","Data":"aecd8f4d2169f0d08391956e528c1d3bf7889f1455066496ede8a9aac32e67b2"}
Sep 14 10:09:14.187154 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:14.187096    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-03fd6d-openshift-default-5854564496-x76m9" podStartSLOduration=1.187074386 podStartE2EDuration="1.187074386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:09:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:09:14.184176578 +0000 UTC m=+1543.223851214" watchObservedRunningTime="2026-09-14 10:09:14.187074386 +0000 UTC m=+1543.226749008"
Sep 14 10:09:14.473535 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:14.473429    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-03fd6d-openshift-default-5854564496-x76m9"
Sep 14 10:09:14.479760 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:14.479726    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-03fd6d-openshift-default-5854564496-x76m9"
Sep 14 10:09:14.534735 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:14.534699    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:09:15.171926 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:15.171887    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-03fd6d-openshift-default-5854564496-x76m9"
Sep 14 10:09:15.173098 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:15.173076    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-03fd6d-openshift-default-5854564496-x76m9"
Sep 14 10:09:15.521171 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:15.521134    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:09:16.074953 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:16.074887    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:09:16.167987 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:16.167941    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:09:16.322242 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:16.322208    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:09:17.493978 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:17.493941    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:09:18.428167 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:18.428124    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:09:18.527405 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:18.527353    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:09:18.935298 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:18.935263    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:09:19.038648 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:19.038593    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:09:19.115346 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:19.115315    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:09:19.788252 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:19.788218    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:09:20.608154 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:20.608107    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:09:21.511180 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:21.511144    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:09:22.530407 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:22.530374    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:09:23.293434 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:23.293374    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:09:23.989329 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:23.989288    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:09:30.154107 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:30.154065    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:09:31.816984 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:31.816946    2563 scope.go:117] "RemoveContainer" containerID="134a848791882da7d32205c1caad532204b88096609f6345e3b871cc6864ea96"
Sep 14 10:09:40.806745 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:40.806707    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:09:41.001924 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:41.001884    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:09:41.526720 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:41.526682    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:09:42.363706 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:42.363674    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:09:43.223198 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:43.223166    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:09:43.765764 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:43.765719    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-29794e-9dnk4"]
Sep 14 10:09:43.770871 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:43.770843    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-29794e-9dnk4"
Sep 14 10:09:43.773310 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:43.773269    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 14 10:09:43.773883 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:43.773836    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gf4xj\""
Sep 14 10:09:43.778198 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:43.778161    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-29794e-9dnk4"]
Sep 14 10:09:43.819583 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:43.819532    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0277bd6e-e6f0-4e66-a59e-3203a393eaec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-29794e-9dnk4\" (UID: \"0277bd6e-e6f0-4e66-a59e-3203a393eaec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-29794e-9dnk4"
Sep 14 10:09:43.920740 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:43.920697    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0277bd6e-e6f0-4e66-a59e-3203a393eaec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-29794e-9dnk4\" (UID: \"0277bd6e-e6f0-4e66-a59e-3203a393eaec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-29794e-9dnk4"
Sep 14 10:09:43.921407 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:43.921388    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0277bd6e-e6f0-4e66-a59e-3203a393eaec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-29794e-9dnk4\" (UID: \"0277bd6e-e6f0-4e66-a59e-3203a393eaec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-29794e-9dnk4"
Sep 14 10:09:44.086162 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:44.086066    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-29794e-9dnk4"
Sep 14 10:09:44.205349 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:44.205307    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:09:44.362604 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:44.362572    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-29794e-9dnk4"]
Sep 14 10:09:44.363797 ip-10-0-132-4 kubenswrapper[2563]: W0914 10:09:44.363769    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0277bd6e_e6f0_4e66_a59e_3203a393eaec.slice/crio-51085f09cabf3365c4cba74b87b3fec10ede47968a8ff6db4a7372070b43f555 WatchSource:0}: Error finding container 51085f09cabf3365c4cba74b87b3fec10ede47968a8ff6db4a7372070b43f555: Status 404 returned error can't find the container with id 51085f09cabf3365c4cba74b87b3fec10ede47968a8ff6db4a7372070b43f555
Sep 14 10:09:44.381609 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:44.381573    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-29794e-9dnk4" event={"ID":"0277bd6e-e6f0-4e66-a59e-3203a393eaec","Type":"ContainerStarted","Data":"51085f09cabf3365c4cba74b87b3fec10ede47968a8ff6db4a7372070b43f555"}
Sep 14 10:09:45.006807 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:45.006763    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:09:45.389757 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:45.389666    2563 generic.go:356] "Generic (PLEG): container finished" podID="0277bd6e-e6f0-4e66-a59e-3203a393eaec" containerID="9c510680245c45740bf170ee13ecf586dc1580fc533836313fc0d1c440c644c6" exitCode=0
Sep 14 10:09:45.389919 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:45.389749    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-29794e-9dnk4" event={"ID":"0277bd6e-e6f0-4e66-a59e-3203a393eaec","Type":"ContainerDied","Data":"9c510680245c45740bf170ee13ecf586dc1580fc533836313fc0d1c440c644c6"}
Sep 14 10:09:45.706073 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:45.705903    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:09:46.516785 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:46.516745    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:09:46.574101 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:46.573848    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-29794e-9dnk4"
Sep 14 10:09:46.652520 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:46.652406    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0277bd6e-e6f0-4e66-a59e-3203a393eaec-maas-api-service-ca\") pod \"0277bd6e-e6f0-4e66-a59e-3203a393eaec\" (UID: \"0277bd6e-e6f0-4e66-a59e-3203a393eaec\") "
Sep 14 10:09:46.652831 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:46.652802    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0277bd6e-e6f0-4e66-a59e-3203a393eaec-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0277bd6e-e6f0-4e66-a59e-3203a393eaec" (UID: "0277bd6e-e6f0-4e66-a59e-3203a393eaec"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:09:46.753914 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:46.753862    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0277bd6e-e6f0-4e66-a59e-3203a393eaec-maas-api-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:09:47.336816 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:47.336779    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:09:47.404696 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:47.404661    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-29794e-9dnk4"
Sep 14 10:09:47.404696 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:47.404680    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-29794e-9dnk4" event={"ID":"0277bd6e-e6f0-4e66-a59e-3203a393eaec","Type":"ContainerDied","Data":"51085f09cabf3365c4cba74b87b3fec10ede47968a8ff6db4a7372070b43f555"}
Sep 14 10:09:47.404912 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:47.404733    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51085f09cabf3365c4cba74b87b3fec10ede47968a8ff6db4a7372070b43f555"
Sep 14 10:09:48.235562 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:48.235523    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:09:49.435658 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:49.435614    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:09:50.461875 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:50.461829    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:09:51.396636 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:51.396588    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:09:54.003423 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:54.003387    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:09:57.491847 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:57.491808    2563 generic.go:356] "Generic (PLEG): container finished" podID="3650d5df-8b7e-465f-83c2-65d3daecd3de" containerID="0d4a420373f7d148687d32fe245e7f2b7076632a6592f2ce61a42e9dd8643767" exitCode=0
Sep 14 10:09:57.492311 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:57.491862    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-03fd6d-openshift-default-5854564496-x76m9" event={"ID":"3650d5df-8b7e-465f-83c2-65d3daecd3de","Type":"ContainerDied","Data":"0d4a420373f7d148687d32fe245e7f2b7076632a6592f2ce61a42e9dd8643767"}
Sep 14 10:09:57.492358 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:57.492329    2563 scope.go:117] "RemoveContainer" containerID="0d4a420373f7d148687d32fe245e7f2b7076632a6592f2ce61a42e9dd8643767"
Sep 14 10:09:57.493678 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:57.493644    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 10:09:57.493787 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:57.493723    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 10:09:57.493787 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:57.493757    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 10:09:58.500773 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:58.500734    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-03fd6d-openshift-default-5854564496-x76m9" event={"ID":"3650d5df-8b7e-465f-83c2-65d3daecd3de","Type":"ContainerStarted","Data":"17b1b6fa9a0bfb64c43ee439bbde2b713ef7a8fd3531a9a039925fcf4346dc86"}
Sep 14 10:09:59.473317 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:59.473268    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-03fd6d-openshift-default-5854564496-x76m9"
Sep 14 10:09:59.474659 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:59.474626    2563 patch_prober.go:28] interesting pod/e2e-api-b-03fd6d-openshift-default-5854564496-x76m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Sep 14 10:09:59.474813 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:59.474685    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-03fd6d-openshift-default-5854564496-x76m9" podUID="3650d5df-8b7e-465f-83c2-65d3daecd3de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Sep 14 10:09:59.646169 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:59.646132    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:09:59.712122 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:59.712091    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:09:59.771255 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:59.771222    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:09:59.853291 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:59.853254    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-29794e-9dnk4"]
Sep 14 10:09:59.854555 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:59.854527    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:09:59.856688 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:59.856658    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-29794e-9dnk4"]
Sep 14 10:09:59.913939 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:09:59.913881    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:00.172381 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:00.172267    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-03fd6d-openshift-default-5854564496-x76m9"
Sep 14 10:10:00.197867 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:00.197831    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:00.345629 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:00.345586    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:00.474251 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:00.474158    2563 patch_prober.go:28] interesting pod/e2e-api-b-03fd6d-openshift-default-5854564496-x76m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Sep 14 10:10:00.474251 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:00.474234    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-03fd6d-openshift-default-5854564496-x76m9" podUID="3650d5df-8b7e-465f-83c2-65d3daecd3de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Sep 14 10:10:01.132364 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:01.132303    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:01.474590 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:01.474496    2563 patch_prober.go:28] interesting pod/e2e-api-b-03fd6d-openshift-default-5854564496-x76m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Sep 14 10:10:01.474590 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:01.474575    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-03fd6d-openshift-default-5854564496-x76m9" podUID="3650d5df-8b7e-465f-83c2-65d3daecd3de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Sep 14 10:10:01.482901 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:01.482867    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0277bd6e-e6f0-4e66-a59e-3203a393eaec" path="/var/lib/kubelet/pods/0277bd6e-e6f0-4e66-a59e-3203a393eaec/volumes"
Sep 14 10:10:01.929641 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:01.929597    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:02.474314 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:02.474272    2563 patch_prober.go:28] interesting pod/e2e-api-b-03fd6d-openshift-default-5854564496-x76m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Sep 14 10:10:02.474969 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:02.474930    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-03fd6d-openshift-default-5854564496-x76m9" podUID="3650d5df-8b7e-465f-83c2-65d3daecd3de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Sep 14 10:10:02.800027 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:02.799969    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:03.474403 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:03.474353    2563 patch_prober.go:28] interesting pod/e2e-api-b-03fd6d-openshift-default-5854564496-x76m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Sep 14 10:10:03.474951 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:03.474409    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-03fd6d-openshift-default-5854564496-x76m9" podUID="3650d5df-8b7e-465f-83c2-65d3daecd3de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Sep 14 10:10:03.733967 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:03.733844    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:04.417170 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:04.417136    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:04.474284 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:04.474243    2563 patch_prober.go:28] interesting pod/e2e-api-b-03fd6d-openshift-default-5854564496-x76m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Sep 14 10:10:04.474469 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:04.474304    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-03fd6d-openshift-default-5854564496-x76m9" podUID="3650d5df-8b7e-465f-83c2-65d3daecd3de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Sep 14 10:10:05.286086 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:05.286023    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:05.483558 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:05.483528    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-03fd6d-openshift-default-5854564496-x76m9"
Sep 14 10:10:05.484922 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:05.484897    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-03fd6d-openshift-default-5854564496-x76m9"
Sep 14 10:10:06.245603 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:06.245570    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:07.218805 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:07.218767    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:07.282802 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:07.282762    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:07.467828 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:07.467772    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:07.780477 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:07.780433    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-03fd6d-tjdts"]
Sep 14 10:10:07.780870 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:07.780855    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0277bd6e-e6f0-4e66-a59e-3203a393eaec" containerName="revoke-keys"
Sep 14 10:10:07.780870 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:07.780870    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0277bd6e-e6f0-4e66-a59e-3203a393eaec" containerName="revoke-keys"
Sep 14 10:10:07.780995 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:07.780948    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0277bd6e-e6f0-4e66-a59e-3203a393eaec" containerName="revoke-keys"
Sep 14 10:10:07.784281 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:07.784252    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-03fd6d-tjdts"
Sep 14 10:10:07.786193 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:07.786166    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 14 10:10:07.786193 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:07.786187    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gf4xj\""
Sep 14 10:10:07.811791 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:07.811754    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8ed006e-f8a4-4343-9291-aa2a627a4a87-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-03fd6d-tjdts\" (UID: \"c8ed006e-f8a4-4343-9291-aa2a627a4a87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-03fd6d-tjdts"
Sep 14 10:10:07.881613 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:07.881575    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-03fd6d-tjdts"]
Sep 14 10:10:07.912604 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:07.912548    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8ed006e-f8a4-4343-9291-aa2a627a4a87-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-03fd6d-tjdts\" (UID: \"c8ed006e-f8a4-4343-9291-aa2a627a4a87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-03fd6d-tjdts"
Sep 14 10:10:07.913325 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:07.913302    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8ed006e-f8a4-4343-9291-aa2a627a4a87-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-03fd6d-tjdts\" (UID: \"c8ed006e-f8a4-4343-9291-aa2a627a4a87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-03fd6d-tjdts"
Sep 14 10:10:08.097410 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:08.097306    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-03fd6d-tjdts"
Sep 14 10:10:08.375598 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:08.375567    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-03fd6d-tjdts"]
Sep 14 10:10:08.376889 ip-10-0-132-4 kubenswrapper[2563]: W0914 10:10:08.376860    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8ed006e_f8a4_4343_9291_aa2a627a4a87.slice/crio-f7415adebc7e278b18c8f70221b80ced6767463519d863f670c0350520a0cd21 WatchSource:0}: Error finding container f7415adebc7e278b18c8f70221b80ced6767463519d863f670c0350520a0cd21: Status 404 returned error can't find the container with id f7415adebc7e278b18c8f70221b80ced6767463519d863f670c0350520a0cd21
Sep 14 10:10:08.583280 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:08.583239    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:08.586354 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:08.586323    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-03fd6d-tjdts" event={"ID":"c8ed006e-f8a4-4343-9291-aa2a627a4a87","Type":"ContainerStarted","Data":"f7415adebc7e278b18c8f70221b80ced6767463519d863f670c0350520a0cd21"}
Sep 14 10:10:08.735745 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:08.735705    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b6812e-rn7mc"]
Sep 14 10:10:08.739361 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:08.739335    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b6812e-rn7mc"
Sep 14 10:10:08.747207 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:08.747116    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b6812e-rn7mc"]
Sep 14 10:10:08.822358 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:08.822321    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/325b71ef-9690-4462-978d-3fc0aeb0d8fc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-b6812e-rn7mc\" (UID: \"325b71ef-9690-4462-978d-3fc0aeb0d8fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b6812e-rn7mc"
Sep 14 10:10:08.860718 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:08.860682    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:08.923497 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:08.923455    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/325b71ef-9690-4462-978d-3fc0aeb0d8fc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-b6812e-rn7mc\" (UID: \"325b71ef-9690-4462-978d-3fc0aeb0d8fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b6812e-rn7mc"
Sep 14 10:10:08.924124 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:08.924105    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/325b71ef-9690-4462-978d-3fc0aeb0d8fc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-b6812e-rn7mc\" (UID: \"325b71ef-9690-4462-978d-3fc0aeb0d8fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b6812e-rn7mc"
Sep 14 10:10:09.054547 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:09.054505    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b6812e-rn7mc"
Sep 14 10:10:09.339979 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:09.339944    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b6812e-rn7mc"]
Sep 14 10:10:09.341899 ip-10-0-132-4 kubenswrapper[2563]: W0914 10:10:09.341863    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod325b71ef_9690_4462_978d_3fc0aeb0d8fc.slice/crio-2ae523b77c3890cb40b42c4a7b955537cc5e236c1df3a785810f09e903259f3f WatchSource:0}: Error finding container 2ae523b77c3890cb40b42c4a7b955537cc5e236c1df3a785810f09e903259f3f: Status 404 returned error can't find the container with id 2ae523b77c3890cb40b42c4a7b955537cc5e236c1df3a785810f09e903259f3f
Sep 14 10:10:09.594566 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:09.594466    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b6812e-rn7mc" event={"ID":"325b71ef-9690-4462-978d-3fc0aeb0d8fc","Type":"ContainerStarted","Data":"2ae523b77c3890cb40b42c4a7b955537cc5e236c1df3a785810f09e903259f3f"}
Sep 14 10:10:09.596468 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:09.596436    2563 generic.go:356] "Generic (PLEG): container finished" podID="c8ed006e-f8a4-4343-9291-aa2a627a4a87" containerID="e6224d6bc0c9b22d9c9084532dbff9dfa10a23ef8dcc03f8e7e1d2d706a10474" exitCode=0
Sep 14 10:10:09.596607 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:09.596474    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-03fd6d-tjdts" event={"ID":"c8ed006e-f8a4-4343-9291-aa2a627a4a87","Type":"ContainerDied","Data":"e6224d6bc0c9b22d9c9084532dbff9dfa10a23ef8dcc03f8e7e1d2d706a10474"}
Sep 14 10:10:09.685651 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:09.685615    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:10.461254 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:10.461156    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:10.605650 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:10.605611    2563 generic.go:356] "Generic (PLEG): container finished" podID="325b71ef-9690-4462-978d-3fc0aeb0d8fc" containerID="345a49e0a3814fdf2eab38ac04b53e4655f39cfe1bbd21a72452635c7815bf45" exitCode=0
Sep 14 10:10:10.606159 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:10.605697    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b6812e-rn7mc" event={"ID":"325b71ef-9690-4462-978d-3fc0aeb0d8fc","Type":"ContainerDied","Data":"345a49e0a3814fdf2eab38ac04b53e4655f39cfe1bbd21a72452635c7815bf45"}
Sep 14 10:10:10.755036 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:10.754984    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-03fd6d-tjdts"
Sep 14 10:10:10.841342 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:10.841296    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8ed006e-f8a4-4343-9291-aa2a627a4a87-maas-api-service-ca\") pod \"c8ed006e-f8a4-4343-9291-aa2a627a4a87\" (UID: \"c8ed006e-f8a4-4343-9291-aa2a627a4a87\") "
Sep 14 10:10:10.841687 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:10.841663    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8ed006e-f8a4-4343-9291-aa2a627a4a87-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c8ed006e-f8a4-4343-9291-aa2a627a4a87" (UID: "c8ed006e-f8a4-4343-9291-aa2a627a4a87"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:10:10.942503 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:10.942454    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8ed006e-f8a4-4343-9291-aa2a627a4a87-maas-api-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:10:11.183572 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:11.183533    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:11.614729 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:11.614691    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-03fd6d-tjdts"
Sep 14 10:10:11.615196 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:11.614724    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-03fd6d-tjdts" event={"ID":"c8ed006e-f8a4-4343-9291-aa2a627a4a87","Type":"ContainerDied","Data":"f7415adebc7e278b18c8f70221b80ced6767463519d863f670c0350520a0cd21"}
Sep 14 10:10:11.615196 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:11.614779    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7415adebc7e278b18c8f70221b80ced6767463519d863f670c0350520a0cd21"
Sep 14 10:10:11.763321 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:11.763240    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b6812e-rn7mc"
Sep 14 10:10:11.851419 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:11.851375    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/325b71ef-9690-4462-978d-3fc0aeb0d8fc-maas-api-service-ca\") pod \"325b71ef-9690-4462-978d-3fc0aeb0d8fc\" (UID: \"325b71ef-9690-4462-978d-3fc0aeb0d8fc\") "
Sep 14 10:10:11.851772 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:11.851747    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/325b71ef-9690-4462-978d-3fc0aeb0d8fc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "325b71ef-9690-4462-978d-3fc0aeb0d8fc" (UID: "325b71ef-9690-4462-978d-3fc0aeb0d8fc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:10:11.952691 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:11.952643    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/325b71ef-9690-4462-978d-3fc0aeb0d8fc-maas-api-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:10:12.150233 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:12.150148    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:12.623205 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:12.623176    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b6812e-rn7mc"
Sep 14 10:10:12.623630 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:12.623215    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b6812e-rn7mc" event={"ID":"325b71ef-9690-4462-978d-3fc0aeb0d8fc","Type":"ContainerDied","Data":"2ae523b77c3890cb40b42c4a7b955537cc5e236c1df3a785810f09e903259f3f"}
Sep 14 10:10:12.623630 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:12.623273    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ae523b77c3890cb40b42c4a7b955537cc5e236c1df3a785810f09e903259f3f"
Sep 14 10:10:13.244907 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:13.244875    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:13.581157 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:13.581050    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b6812e/distinct-2-w0-b6812e-kserve-7ffbdbf7d-kx89t"]
Sep 14 10:10:13.595119 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:13.595076    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b6812e/distinct-w0-b6812e-kserve-786cffbfd5-q74np"]
Sep 14 10:10:13.667730 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:13.667684    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b6812e/simulator-w0-b6812e-kserve-6c5dd8795-rvbtb"]
Sep 14 10:10:13.738663 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:13.738629    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:14.202471 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:14.202433    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:14.644266 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:14.644229    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:15.002709 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:15.002664    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:15.909438 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:15.909403    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:16.608508 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:16.608460    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:18.102062 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:18.101963    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:18.928934 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:18.928879    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:23.770942 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:23.770848    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:23.879482 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:23.879448    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:23.977649 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:23.977610    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:24.039048 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:24.038931    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-03fd6d-tjdts"]
Sep 14 10:10:24.046622 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:24.046582    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-03fd6d-tjdts"]
Sep 14 10:10:24.056508 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:24.056472    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:24.369029 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:24.368923    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:24.482220 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:24.482184    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:25.338396 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:25.338363    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:25.483208 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:25.483171    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8ed006e-f8a4-4343-9291-aa2a627a4a87" path="/var/lib/kubelet/pods/c8ed006e-f8a4-4343-9291-aa2a627a4a87/volumes"
Sep 14 10:10:26.205800 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:26.205749    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:27.221436 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:27.221398    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:27.321664 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:27.321632    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:27.421043 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:27.420982    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2901ec-9krtq"]
Sep 14 10:10:27.421474 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:27.421460    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8ed006e-f8a4-4343-9291-aa2a627a4a87" containerName="revoke-keys"
Sep 14 10:10:27.421518 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:27.421476    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8ed006e-f8a4-4343-9291-aa2a627a4a87" containerName="revoke-keys"
Sep 14 10:10:27.421518 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:27.421495    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="325b71ef-9690-4462-978d-3fc0aeb0d8fc" containerName="revoke-keys"
Sep 14 10:10:27.421518 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:27.421500    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="325b71ef-9690-4462-978d-3fc0aeb0d8fc" containerName="revoke-keys"
Sep 14 10:10:27.421620 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:27.421562    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="325b71ef-9690-4462-978d-3fc0aeb0d8fc" containerName="revoke-keys"
Sep 14 10:10:27.421620 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:27.421573    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8ed006e-f8a4-4343-9291-aa2a627a4a87" containerName="revoke-keys"
Sep 14 10:10:27.425201 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:27.425177    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2901ec-9krtq"
Sep 14 10:10:27.427549 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:27.427519    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gf4xj\""
Sep 14 10:10:27.427700 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:27.427564    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 14 10:10:27.435140 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:27.435108    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2901ec-9krtq"]
Sep 14 10:10:27.521192 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:27.521150    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8c2b453-d296-45c9-a279-0ff272cc3212-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-2901ec-9krtq\" (UID: \"f8c2b453-d296-45c9-a279-0ff272cc3212\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2901ec-9krtq"
Sep 14 10:10:27.621987 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:27.621944    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8c2b453-d296-45c9-a279-0ff272cc3212-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-2901ec-9krtq\" (UID: \"f8c2b453-d296-45c9-a279-0ff272cc3212\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2901ec-9krtq"
Sep 14 10:10:27.622677 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:27.622654    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8c2b453-d296-45c9-a279-0ff272cc3212-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-2901ec-9krtq\" (UID: \"f8c2b453-d296-45c9-a279-0ff272cc3212\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2901ec-9krtq"
Sep 14 10:10:27.738994 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:27.738950    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2901ec-9krtq"
Sep 14 10:10:28.024721 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:28.024682    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2901ec-9krtq"]
Sep 14 10:10:28.026033 ip-10-0-132-4 kubenswrapper[2563]: W0914 10:10:28.025973    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8c2b453_d296_45c9_a279_0ff272cc3212.slice/crio-8f2b0bf50c7c0f95cba49c39cf440f8378957ca5708dda207f9e31ab42e7571b WatchSource:0}: Error finding container 8f2b0bf50c7c0f95cba49c39cf440f8378957ca5708dda207f9e31ab42e7571b: Status 404 returned error can't find the container with id 8f2b0bf50c7c0f95cba49c39cf440f8378957ca5708dda207f9e31ab42e7571b
Sep 14 10:10:28.224129 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:28.224093    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:28.602639 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:28.602544    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-b6812e/distinct-2-w0-b6812e-kserve-7ffbdbf7d-kx89t" podUID="dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2" containerName="main" containerID="cri-o://8fbd676b89d67b475bd7b4b13026f6a6bba136393d4bcc60b6997f6c957aea73" gracePeriod=45
Sep 14 10:10:28.615882 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:28.615834    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-b6812e/distinct-w0-b6812e-kserve-786cffbfd5-q74np" podUID="60fe835c-e5f9-44ae-9dbc-eff55a5bc50a" containerName="main" containerID="cri-o://195a48f81b5af2399f25f6ef37137f2cf814c3dcb985649efd845f04e0fcded5" gracePeriod=45
Sep 14 10:10:28.688067 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:28.687991    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-b6812e/simulator-w0-b6812e-kserve-6c5dd8795-rvbtb" podUID="c6fd0d32-d8ff-411e-8183-502161d51def" containerName="main" containerID="cri-o://6b6fd648822e3e5ffdd9c6baae0d4e6b23f97f2c0b7f7ffd28bd5f3148c52e69" gracePeriod=45
Sep 14 10:10:28.736937 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:28.736901    2563 generic.go:356] "Generic (PLEG): container finished" podID="60fe835c-e5f9-44ae-9dbc-eff55a5bc50a" containerID="195a48f81b5af2399f25f6ef37137f2cf814c3dcb985649efd845f04e0fcded5" exitCode=0
Sep 14 10:10:28.737139 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:28.736978    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b6812e/distinct-w0-b6812e-kserve-786cffbfd5-q74np" event={"ID":"60fe835c-e5f9-44ae-9dbc-eff55a5bc50a","Type":"ContainerDied","Data":"195a48f81b5af2399f25f6ef37137f2cf814c3dcb985649efd845f04e0fcded5"}
Sep 14 10:10:28.738743 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:28.738711    2563 generic.go:356] "Generic (PLEG): container finished" podID="dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2" containerID="8fbd676b89d67b475bd7b4b13026f6a6bba136393d4bcc60b6997f6c957aea73" exitCode=0
Sep 14 10:10:28.738882 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:28.738779    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b6812e/distinct-2-w0-b6812e-kserve-7ffbdbf7d-kx89t" event={"ID":"dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2","Type":"ContainerDied","Data":"8fbd676b89d67b475bd7b4b13026f6a6bba136393d4bcc60b6997f6c957aea73"}
Sep 14 10:10:28.740595 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:28.740566    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2901ec-9krtq" event={"ID":"f8c2b453-d296-45c9-a279-0ff272cc3212","Type":"ContainerStarted","Data":"ef2510bd99ebfa87d7766b0a5cbe5a5916d5e78bc001f4db8ea3ab7efd5f2153"}
Sep 14 10:10:28.740743 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:28.740615    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2901ec-9krtq" event={"ID":"f8c2b453-d296-45c9-a279-0ff272cc3212","Type":"ContainerStarted","Data":"8f2b0bf50c7c0f95cba49c39cf440f8378957ca5708dda207f9e31ab42e7571b"}
Sep 14 10:10:28.757229 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:28.757172    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2901ec-9krtq" podStartSLOduration=1.757152794 podStartE2EDuration="1.757152794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:10:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:10:28.754922985 +0000 UTC m=+1617.794597607" watchObservedRunningTime="2026-09-14 10:10:28.757152794 +0000 UTC m=+1617.796827415"
Sep 14 10:10:28.869143 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:28.868355    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:28.878721 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:28.878683    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6812e-29823000-g4fqs"]
Sep 14 10:10:28.883840 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:28.883793    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6812e-29823000-g4fqs"]
Sep 14 10:10:28.960352 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:28.960313    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:29.004409 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.004376    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b6812e/distinct-w0-b6812e-kserve-786cffbfd5-q74np"
Sep 14 10:10:29.035553 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.035517    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:29.052292 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.052240    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b6812e/distinct-2-w0-b6812e-kserve-7ffbdbf7d-kx89t"
Sep 14 10:10:29.056101 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.056045    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b6812e/simulator-w0-b6812e-kserve-6c5dd8795-rvbtb"
Sep 14 10:10:29.097676 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.097637    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:29.139664 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.139569    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6fd0d32-d8ff-411e-8183-502161d51def-dshm\") pod \"c6fd0d32-d8ff-411e-8183-502161d51def\" (UID: \"c6fd0d32-d8ff-411e-8183-502161d51def\") "
Sep 14 10:10:29.139664 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.139648    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2-home\") pod \"dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2\" (UID: \"dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2\") "
Sep 14 10:10:29.139893 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.139676    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60fe835c-e5f9-44ae-9dbc-eff55a5bc50a-tmp-dir\") pod \"60fe835c-e5f9-44ae-9dbc-eff55a5bc50a\" (UID: \"60fe835c-e5f9-44ae-9dbc-eff55a5bc50a\") "
Sep 14 10:10:29.139893 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.139712    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2-model-cache\") pod \"dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2\" (UID: \"dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2\") "
Sep 14 10:10:29.139893 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.139747    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60fe835c-e5f9-44ae-9dbc-eff55a5bc50a-model-cache\") pod \"60fe835c-e5f9-44ae-9dbc-eff55a5bc50a\" (UID: \"60fe835c-e5f9-44ae-9dbc-eff55a5bc50a\") "
Sep 14 10:10:29.139893 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.139775    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6fd0d32-d8ff-411e-8183-502161d51def-tls-certs\") pod \"c6fd0d32-d8ff-411e-8183-502161d51def\" (UID: \"c6fd0d32-d8ff-411e-8183-502161d51def\") "
Sep 14 10:10:29.139893 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.139809    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6fd0d32-d8ff-411e-8183-502161d51def-tmp-dir\") pod \"c6fd0d32-d8ff-411e-8183-502161d51def\" (UID: \"c6fd0d32-d8ff-411e-8183-502161d51def\") "
Sep 14 10:10:29.139893 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.139848    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60fe835c-e5f9-44ae-9dbc-eff55a5bc50a-tls-certs\") pod \"60fe835c-e5f9-44ae-9dbc-eff55a5bc50a\" (UID: \"60fe835c-e5f9-44ae-9dbc-eff55a5bc50a\") "
Sep 14 10:10:29.140296 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.139901    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4ft8c\" (UniqueName: \"kubernetes.io/projected/dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2-kube-api-access-4ft8c\") pod \"dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2\" (UID: \"dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2\") "
Sep 14 10:10:29.140296 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.139928    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2-tls-certs\") pod \"dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2\" (UID: \"dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2\") "
Sep 14 10:10:29.140296 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.139962    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2-home" (OuterVolumeSpecName: "home") pod "dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2" (UID: "dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:10:29.140296 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.140029    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60fe835c-e5f9-44ae-9dbc-eff55a5bc50a-model-cache" (OuterVolumeSpecName: "model-cache") pod "60fe835c-e5f9-44ae-9dbc-eff55a5bc50a" (UID: "60fe835c-e5f9-44ae-9dbc-eff55a5bc50a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:10:29.140296 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.140098    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2-model-cache" (OuterVolumeSpecName: "model-cache") pod "dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2" (UID: "dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:10:29.140296 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.140186    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6fd0d32-d8ff-411e-8183-502161d51def-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c6fd0d32-d8ff-411e-8183-502161d51def" (UID: "c6fd0d32-d8ff-411e-8183-502161d51def"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:10:29.140296 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.140252    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60fe835c-e5f9-44ae-9dbc-eff55a5bc50a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "60fe835c-e5f9-44ae-9dbc-eff55a5bc50a" (UID: "60fe835c-e5f9-44ae-9dbc-eff55a5bc50a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:10:29.140657 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.140375    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6fd0d32-d8ff-411e-8183-502161d51def-home\") pod \"c6fd0d32-d8ff-411e-8183-502161d51def\" (UID: \"c6fd0d32-d8ff-411e-8183-502161d51def\") "
Sep 14 10:10:29.140657 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.140410    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2-tmp-dir\") pod \"dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2\" (UID: \"dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2\") "
Sep 14 10:10:29.140657 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.140442    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6fd0d32-d8ff-411e-8183-502161d51def-model-cache\") pod \"c6fd0d32-d8ff-411e-8183-502161d51def\" (UID: \"c6fd0d32-d8ff-411e-8183-502161d51def\") "
Sep 14 10:10:29.140657 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.140475    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60fe835c-e5f9-44ae-9dbc-eff55a5bc50a-dshm\") pod \"60fe835c-e5f9-44ae-9dbc-eff55a5bc50a\" (UID: \"60fe835c-e5f9-44ae-9dbc-eff55a5bc50a\") "
Sep 14 10:10:29.140657 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.140517    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60fe835c-e5f9-44ae-9dbc-eff55a5bc50a-home\") pod \"60fe835c-e5f9-44ae-9dbc-eff55a5bc50a\" (UID: \"60fe835c-e5f9-44ae-9dbc-eff55a5bc50a\") "
Sep 14 10:10:29.140657 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.140553    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rz22f\" (UniqueName: \"kubernetes.io/projected/60fe835c-e5f9-44ae-9dbc-eff55a5bc50a-kube-api-access-rz22f\") pod \"60fe835c-e5f9-44ae-9dbc-eff55a5bc50a\" (UID: \"60fe835c-e5f9-44ae-9dbc-eff55a5bc50a\") "
Sep 14 10:10:29.140657 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.140587    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2-dshm\") pod \"dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2\" (UID: \"dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2\") "
Sep 14 10:10:29.140657 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.140635    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ndhs\" (UniqueName: \"kubernetes.io/projected/c6fd0d32-d8ff-411e-8183-502161d51def-kube-api-access-2ndhs\") pod \"c6fd0d32-d8ff-411e-8183-502161d51def\" (UID: \"c6fd0d32-d8ff-411e-8183-502161d51def\") "
Sep 14 10:10:29.141168 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.140654    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6fd0d32-d8ff-411e-8183-502161d51def-home" (OuterVolumeSpecName: "home") pod "c6fd0d32-d8ff-411e-8183-502161d51def" (UID: "c6fd0d32-d8ff-411e-8183-502161d51def"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:10:29.141168 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.140747    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6fd0d32-d8ff-411e-8183-502161d51def-model-cache" (OuterVolumeSpecName: "model-cache") pod "c6fd0d32-d8ff-411e-8183-502161d51def" (UID: "c6fd0d32-d8ff-411e-8183-502161d51def"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:10:29.141168 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.140768    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2" (UID: "dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:10:29.141168 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.141127    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2-model-cache\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:10:29.141168 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.141150    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60fe835c-e5f9-44ae-9dbc-eff55a5bc50a-model-cache\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:10:29.141168 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.141162    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6fd0d32-d8ff-411e-8183-502161d51def-tmp-dir\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:10:29.141531 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.141174    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6fd0d32-d8ff-411e-8183-502161d51def-home\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:10:29.141531 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.141186    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2-tmp-dir\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:10:29.141531 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.141197    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6fd0d32-d8ff-411e-8183-502161d51def-model-cache\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:10:29.141531 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.141210    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2-home\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:10:29.141531 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.141222    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60fe835c-e5f9-44ae-9dbc-eff55a5bc50a-tmp-dir\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:10:29.141531 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.141419    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60fe835c-e5f9-44ae-9dbc-eff55a5bc50a-home" (OuterVolumeSpecName: "home") pod "60fe835c-e5f9-44ae-9dbc-eff55a5bc50a" (UID: "60fe835c-e5f9-44ae-9dbc-eff55a5bc50a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:10:29.142528 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.142505    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6fd0d32-d8ff-411e-8183-502161d51def-dshm" (OuterVolumeSpecName: "dshm") pod "c6fd0d32-d8ff-411e-8183-502161d51def" (UID: "c6fd0d32-d8ff-411e-8183-502161d51def"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:10:29.143927 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.143898    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2-kube-api-access-4ft8c" (OuterVolumeSpecName: "kube-api-access-4ft8c") pod "dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2" (UID: "dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2"). InnerVolumeSpecName "kube-api-access-4ft8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:10:29.144131 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.143914    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2" (UID: "dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:10:29.144131 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.143985    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60fe835c-e5f9-44ae-9dbc-eff55a5bc50a-dshm" (OuterVolumeSpecName: "dshm") pod "60fe835c-e5f9-44ae-9dbc-eff55a5bc50a" (UID: "60fe835c-e5f9-44ae-9dbc-eff55a5bc50a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:10:29.144131 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.144038    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6fd0d32-d8ff-411e-8183-502161d51def-kube-api-access-2ndhs" (OuterVolumeSpecName: "kube-api-access-2ndhs") pod "c6fd0d32-d8ff-411e-8183-502161d51def" (UID: "c6fd0d32-d8ff-411e-8183-502161d51def"). InnerVolumeSpecName "kube-api-access-2ndhs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:10:29.144131 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.144089    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6fd0d32-d8ff-411e-8183-502161d51def-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c6fd0d32-d8ff-411e-8183-502161d51def" (UID: "c6fd0d32-d8ff-411e-8183-502161d51def"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:10:29.144319 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.144141    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60fe835c-e5f9-44ae-9dbc-eff55a5bc50a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "60fe835c-e5f9-44ae-9dbc-eff55a5bc50a" (UID: "60fe835c-e5f9-44ae-9dbc-eff55a5bc50a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:10:29.144498 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.144478    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60fe835c-e5f9-44ae-9dbc-eff55a5bc50a-kube-api-access-rz22f" (OuterVolumeSpecName: "kube-api-access-rz22f") pod "60fe835c-e5f9-44ae-9dbc-eff55a5bc50a" (UID: "60fe835c-e5f9-44ae-9dbc-eff55a5bc50a"). InnerVolumeSpecName "kube-api-access-rz22f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:10:29.145712 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.145691    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2-dshm" (OuterVolumeSpecName: "dshm") pod "dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2" (UID: "dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:10:29.241940 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.241900    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60fe835c-e5f9-44ae-9dbc-eff55a5bc50a-tls-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:10:29.241940 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.241938    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ft8c\" (UniqueName: \"kubernetes.io/projected/dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2-kube-api-access-4ft8c\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:10:29.242407 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.241953    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2-tls-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:10:29.242407 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.241961    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60fe835c-e5f9-44ae-9dbc-eff55a5bc50a-dshm\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:10:29.242407 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.241969    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60fe835c-e5f9-44ae-9dbc-eff55a5bc50a-home\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:10:29.242407 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.241979    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz22f\" (UniqueName: \"kubernetes.io/projected/60fe835c-e5f9-44ae-9dbc-eff55a5bc50a-kube-api-access-rz22f\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:10:29.242407 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.241987    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2-dshm\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:10:29.242407 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.241995    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ndhs\" (UniqueName: \"kubernetes.io/projected/c6fd0d32-d8ff-411e-8183-502161d51def-kube-api-access-2ndhs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:10:29.242407 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.242025    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6fd0d32-d8ff-411e-8183-502161d51def-dshm\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:10:29.242407 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.242039    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6fd0d32-d8ff-411e-8183-502161d51def-tls-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:10:29.267918 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.267877    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:29.319822 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.319785    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b6812e-rn7mc"]
Sep 14 10:10:29.322722 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.322685    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b6812e-rn7mc"]
Sep 14 10:10:29.461697 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.461560    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:29.484642 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.484548    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="325b71ef-9690-4462-978d-3fc0aeb0d8fc" path="/var/lib/kubelet/pods/325b71ef-9690-4462-978d-3fc0aeb0d8fc/volumes"
Sep 14 10:10:29.484908 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.484883    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d123f30-e7e7-4a5a-add5-73ee74bf4742" path="/var/lib/kubelet/pods/7d123f30-e7e7-4a5a-add5-73ee74bf4742/volumes"
Sep 14 10:10:29.623160 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.618559    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:29.748767 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.748730    2563 generic.go:356] "Generic (PLEG): container finished" podID="c6fd0d32-d8ff-411e-8183-502161d51def" containerID="6b6fd648822e3e5ffdd9c6baae0d4e6b23f97f2c0b7f7ffd28bd5f3148c52e69" exitCode=0
Sep 14 10:10:29.748963 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.748803    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b6812e/simulator-w0-b6812e-kserve-6c5dd8795-rvbtb"
Sep 14 10:10:29.748963 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.748814    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b6812e/simulator-w0-b6812e-kserve-6c5dd8795-rvbtb" event={"ID":"c6fd0d32-d8ff-411e-8183-502161d51def","Type":"ContainerDied","Data":"6b6fd648822e3e5ffdd9c6baae0d4e6b23f97f2c0b7f7ffd28bd5f3148c52e69"}
Sep 14 10:10:29.748963 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.748860    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b6812e/simulator-w0-b6812e-kserve-6c5dd8795-rvbtb" event={"ID":"c6fd0d32-d8ff-411e-8183-502161d51def","Type":"ContainerDied","Data":"42c781a639f159fadc48bb4239cda3fa0696f41e54bf786216e4a1e76db839a7"}
Sep 14 10:10:29.748963 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.748884    2563 scope.go:117] "RemoveContainer" containerID="6b6fd648822e3e5ffdd9c6baae0d4e6b23f97f2c0b7f7ffd28bd5f3148c52e69"
Sep 14 10:10:29.750471 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.750454    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b6812e/distinct-w0-b6812e-kserve-786cffbfd5-q74np"
Sep 14 10:10:29.750574 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.750486    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b6812e/distinct-w0-b6812e-kserve-786cffbfd5-q74np" event={"ID":"60fe835c-e5f9-44ae-9dbc-eff55a5bc50a","Type":"ContainerDied","Data":"0cd2f339b16fddb2c2a35b0ec9d1e2864d2f75250145eacdb09478ef5eebfaff"}
Sep 14 10:10:29.752370 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.752325    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b6812e/distinct-2-w0-b6812e-kserve-7ffbdbf7d-kx89t" event={"ID":"dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2","Type":"ContainerDied","Data":"05e2ec598c6e7be0bc1a45ffe3baacf6435ec567c0d6ebd32423888f1d6c99e8"}
Sep 14 10:10:29.752581 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.752362    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b6812e/distinct-2-w0-b6812e-kserve-7ffbdbf7d-kx89t"
Sep 14 10:10:29.754984 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.754953    2563 generic.go:356] "Generic (PLEG): container finished" podID="f8c2b453-d296-45c9-a279-0ff272cc3212" containerID="ef2510bd99ebfa87d7766b0a5cbe5a5916d5e78bc001f4db8ea3ab7efd5f2153" exitCode=0
Sep 14 10:10:29.755102 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.754992    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2901ec-9krtq" event={"ID":"f8c2b453-d296-45c9-a279-0ff272cc3212","Type":"ContainerDied","Data":"ef2510bd99ebfa87d7766b0a5cbe5a5916d5e78bc001f4db8ea3ab7efd5f2153"}
Sep 14 10:10:29.760734 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.760714    2563 scope.go:117] "RemoveContainer" containerID="6b6fd648822e3e5ffdd9c6baae0d4e6b23f97f2c0b7f7ffd28bd5f3148c52e69"
Sep 14 10:10:29.761049 ip-10-0-132-4 kubenswrapper[2563]: E0914 10:10:29.761027    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b6fd648822e3e5ffdd9c6baae0d4e6b23f97f2c0b7f7ffd28bd5f3148c52e69\": container with ID starting with 6b6fd648822e3e5ffdd9c6baae0d4e6b23f97f2c0b7f7ffd28bd5f3148c52e69 not found: ID does not exist" containerID="6b6fd648822e3e5ffdd9c6baae0d4e6b23f97f2c0b7f7ffd28bd5f3148c52e69"
Sep 14 10:10:29.761138 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.761057    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b6fd648822e3e5ffdd9c6baae0d4e6b23f97f2c0b7f7ffd28bd5f3148c52e69"} err="failed to get container status \"6b6fd648822e3e5ffdd9c6baae0d4e6b23f97f2c0b7f7ffd28bd5f3148c52e69\": rpc error: code = NotFound desc = could not find container \"6b6fd648822e3e5ffdd9c6baae0d4e6b23f97f2c0b7f7ffd28bd5f3148c52e69\": container with ID starting with 6b6fd648822e3e5ffdd9c6baae0d4e6b23f97f2c0b7f7ffd28bd5f3148c52e69 not found: ID does not exist"
Sep 14 10:10:29.761138 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.761076    2563 scope.go:117] "RemoveContainer" containerID="195a48f81b5af2399f25f6ef37137f2cf814c3dcb985649efd845f04e0fcded5"
Sep 14 10:10:29.771126 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.771082    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b6812e/distinct-w0-b6812e-kserve-786cffbfd5-q74np"]
Sep 14 10:10:29.771432 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.771412    2563 scope.go:117] "RemoveContainer" containerID="8fbd676b89d67b475bd7b4b13026f6a6bba136393d4bcc60b6997f6c957aea73"
Sep 14 10:10:29.773924 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.773899    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-b6812e/distinct-w0-b6812e-kserve-786cffbfd5-q74np"]
Sep 14 10:10:29.785029 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.784977    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b6812e/simulator-w0-b6812e-kserve-6c5dd8795-rvbtb"]
Sep 14 10:10:29.788098 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.788070    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-b6812e/simulator-w0-b6812e-kserve-6c5dd8795-rvbtb"]
Sep 14 10:10:29.815259 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.815220    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b6812e/distinct-2-w0-b6812e-kserve-7ffbdbf7d-kx89t"]
Sep 14 10:10:29.820879 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:29.820844    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-b6812e/distinct-2-w0-b6812e-kserve-7ffbdbf7d-kx89t"]
Sep 14 10:10:30.297443 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:30.297408    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:30.913259 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:30.913221    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2901ec-9krtq"
Sep 14 10:10:31.061135 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:31.060994    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8c2b453-d296-45c9-a279-0ff272cc3212-maas-api-service-ca\") pod \"f8c2b453-d296-45c9-a279-0ff272cc3212\" (UID: \"f8c2b453-d296-45c9-a279-0ff272cc3212\") "
Sep 14 10:10:31.061490 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:31.061466    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8c2b453-d296-45c9-a279-0ff272cc3212-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f8c2b453-d296-45c9-a279-0ff272cc3212" (UID: "f8c2b453-d296-45c9-a279-0ff272cc3212"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:10:31.163072 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:31.163030    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8c2b453-d296-45c9-a279-0ff272cc3212-maas-api-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:10:31.271815 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:31.271776    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:31.482979 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:31.482892    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60fe835c-e5f9-44ae-9dbc-eff55a5bc50a" path="/var/lib/kubelet/pods/60fe835c-e5f9-44ae-9dbc-eff55a5bc50a/volumes"
Sep 14 10:10:31.483433 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:31.483415    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6fd0d32-d8ff-411e-8183-502161d51def" path="/var/lib/kubelet/pods/c6fd0d32-d8ff-411e-8183-502161d51def/volumes"
Sep 14 10:10:31.483795 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:31.483782    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2" path="/var/lib/kubelet/pods/dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2/volumes"
Sep 14 10:10:31.718751 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:31.718700    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:31.773624 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:31.773596    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2901ec-9krtq"
Sep 14 10:10:31.773838 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:31.773632    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2901ec-9krtq" event={"ID":"f8c2b453-d296-45c9-a279-0ff272cc3212","Type":"ContainerDied","Data":"8f2b0bf50c7c0f95cba49c39cf440f8378957ca5708dda207f9e31ab42e7571b"}
Sep 14 10:10:31.773838 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:31.773685    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f2b0bf50c7c0f95cba49c39cf440f8378957ca5708dda207f9e31ab42e7571b"
Sep 14 10:10:31.814333 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:31.814292    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:31.842541 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:31.842507    2563 scope.go:117] "RemoveContainer" containerID="e15a95802a047439f8f88fba14f5551479206b8edb85be9081cd8bedae86b378"
Sep 14 10:10:31.852646 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:31.852621    2563 scope.go:117] "RemoveContainer" containerID="341d49e2e204ae07b199775216732718362e567a846d49d95e0005b9ee7e00cb"
Sep 14 10:10:31.862505 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:31.862483    2563 scope.go:117] "RemoveContainer" containerID="aeaab86acb8b1d09a11f965334d95c4812a92b65e48bdfdbf63879aa30b8bf16"
Sep 14 10:10:31.970352 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:31.970297    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-03fd6d-openshift-default-5854564496-x76m9"]
Sep 14 10:10:31.970616 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:31.970589    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-03fd6d-openshift-default-5854564496-x76m9" podUID="3650d5df-8b7e-465f-83c2-65d3daecd3de" containerName="istio-proxy" containerID="cri-o://17b1b6fa9a0bfb64c43ee439bbde2b713ef7a8fd3531a9a039925fcf4346dc86" gracePeriod=30
Sep 14 10:10:32.035660 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:32.035568    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:32.351413 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:32.351294    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-2901ec/distinct-w4-2901ec-kserve-9fc4b5697-xblp8"]
Sep 14 10:10:32.397233 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:32.397194    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:32.405095 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:32.405055    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-2901ec/simulator-w4-2901ec-kserve-7c4cdb8dff-nhllq"]
Sep 14 10:10:32.543179 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:32.543135    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8fd006-ldjrt"]
Sep 14 10:10:32.543579 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:32.543567    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60fe835c-e5f9-44ae-9dbc-eff55a5bc50a" containerName="main"
Sep 14 10:10:32.543579 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:32.543579    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="60fe835c-e5f9-44ae-9dbc-eff55a5bc50a" containerName="main"
Sep 14 10:10:32.543668 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:32.543589    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2" containerName="main"
Sep 14 10:10:32.543668 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:32.543595    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2" containerName="main"
Sep 14 10:10:32.543668 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:32.543606    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6fd0d32-d8ff-411e-8183-502161d51def" containerName="main"
Sep 14 10:10:32.543668 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:32.543611    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6fd0d32-d8ff-411e-8183-502161d51def" containerName="main"
Sep 14 10:10:32.543668 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:32.543618    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8c2b453-d296-45c9-a279-0ff272cc3212" containerName="revoke-keys"
Sep 14 10:10:32.543668 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:32.543624    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8c2b453-d296-45c9-a279-0ff272cc3212" containerName="revoke-keys"
Sep 14 10:10:32.543860 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:32.543686    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="60fe835c-e5f9-44ae-9dbc-eff55a5bc50a" containerName="main"
Sep 14 10:10:32.543860 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:32.543696    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8c2b453-d296-45c9-a279-0ff272cc3212" containerName="revoke-keys"
Sep 14 10:10:32.543860 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:32.543704    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6fd0d32-d8ff-411e-8183-502161d51def" containerName="main"
Sep 14 10:10:32.543860 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:32.543712    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="dedd7da3-0dff-4eb7-bc44-4aac8d9aa5b2" containerName="main"
Sep 14 10:10:32.546890 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:32.546866    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8fd006-ldjrt"
Sep 14 10:10:32.550148 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:32.550115    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gf4xj\""
Sep 14 10:10:32.550737 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:32.550542    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 14 10:10:32.560683 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:32.560650    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8fd006-ldjrt"]
Sep 14 10:10:32.680326 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:32.680206    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05627ca1-2838-4242-bad6-069eae834fc9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-8fd006-ldjrt\" (UID: \"05627ca1-2838-4242-bad6-069eae834fc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8fd006-ldjrt"
Sep 14 10:10:32.690224 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:32.690183    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:32.781524 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:32.781484    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05627ca1-2838-4242-bad6-069eae834fc9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-8fd006-ldjrt\" (UID: \"05627ca1-2838-4242-bad6-069eae834fc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8fd006-ldjrt"
Sep 14 10:10:32.782222 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:32.782199    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05627ca1-2838-4242-bad6-069eae834fc9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-8fd006-ldjrt\" (UID: \"05627ca1-2838-4242-bad6-069eae834fc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8fd006-ldjrt"
Sep 14 10:10:32.860989 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:32.860942    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8fd006-ldjrt"
Sep 14 10:10:33.154052 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:33.154017    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8fd006-ldjrt"]
Sep 14 10:10:33.156317 ip-10-0-132-4 kubenswrapper[2563]: W0914 10:10:33.156283    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05627ca1_2838_4242_bad6_069eae834fc9.slice/crio-732e3b42123d9df0ef12b3a5b80dcfbbe3aa31ed9d92e875c3ec8984d0c825c6 WatchSource:0}: Error finding container 732e3b42123d9df0ef12b3a5b80dcfbbe3aa31ed9d92e875c3ec8984d0c825c6: Status 404 returned error can't find the container with id 732e3b42123d9df0ef12b3a5b80dcfbbe3aa31ed9d92e875c3ec8984d0c825c6
Sep 14 10:10:33.793089 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:33.793044    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8fd006-ldjrt" event={"ID":"05627ca1-2838-4242-bad6-069eae834fc9","Type":"ContainerStarted","Data":"988fe03117d1c3481af1bda1ae9c609b7599706bcc07b4c37994e86093f02fef"}
Sep 14 10:10:33.793522 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:33.793107    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8fd006-ldjrt" event={"ID":"05627ca1-2838-4242-bad6-069eae834fc9","Type":"ContainerStarted","Data":"732e3b42123d9df0ef12b3a5b80dcfbbe3aa31ed9d92e875c3ec8984d0c825c6"}
Sep 14 10:10:33.814572 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:33.814514    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8fd006-ldjrt" podStartSLOduration=1.814493316 podStartE2EDuration="1.814493316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:10:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:10:33.813206528 +0000 UTC m=+1622.852881163" watchObservedRunningTime="2026-09-14 10:10:33.814493316 +0000 UTC m=+1622.854167938"
Sep 14 10:10:33.815444 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:33.815420    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:34.700254 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:34.700210    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:34.801580 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:34.801543    2563 generic.go:356] "Generic (PLEG): container finished" podID="05627ca1-2838-4242-bad6-069eae834fc9" containerID="988fe03117d1c3481af1bda1ae9c609b7599706bcc07b4c37994e86093f02fef" exitCode=0
Sep 14 10:10:34.802052 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:34.801639    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8fd006-ldjrt" event={"ID":"05627ca1-2838-4242-bad6-069eae834fc9","Type":"ContainerDied","Data":"988fe03117d1c3481af1bda1ae9c609b7599706bcc07b4c37994e86093f02fef"}
Sep 14 10:10:35.486270 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:35.486224    2563 patch_prober.go:28] interesting pod/e2e-api-b-03fd6d-openshift-default-5854564496-x76m9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 14 10:10:35.486449 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:35.486292    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-03fd6d-openshift-default-5854564496-x76m9" podUID="3650d5df-8b7e-465f-83c2-65d3daecd3de" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 14 10:10:35.791262 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:35.791166    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:35.948656 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:35.948622    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8fd006-ldjrt"
Sep 14 10:10:36.045039 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:36.044923    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:36.119780 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:36.119737    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05627ca1-2838-4242-bad6-069eae834fc9-maas-api-service-ca\") pod \"05627ca1-2838-4242-bad6-069eae834fc9\" (UID: \"05627ca1-2838-4242-bad6-069eae834fc9\") "
Sep 14 10:10:36.120148 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:36.120124    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05627ca1-2838-4242-bad6-069eae834fc9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "05627ca1-2838-4242-bad6-069eae834fc9" (UID: "05627ca1-2838-4242-bad6-069eae834fc9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:10:36.141350 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:36.141308    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:36.221386 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:36.221340    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05627ca1-2838-4242-bad6-069eae834fc9-maas-api-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:10:36.265367 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:36.265325    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw"]
Sep 14 10:10:36.265628 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:36.265601    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw" podUID="a9d150e6-194e-44be-8a5f-bb826032f664" containerName="istio-proxy" containerID="cri-o://606094d57bf0e58f8d128d7feda6dd7fe2c4919e29a3f387551845738a77ac18" gracePeriod=30
Sep 14 10:10:36.276926 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:36.276889    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:36.628511 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:36.628473    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:36.817289 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:36.817252    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8fd006-ldjrt"
Sep 14 10:10:36.817289 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:36.817282    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8fd006-ldjrt" event={"ID":"05627ca1-2838-4242-bad6-069eae834fc9","Type":"ContainerDied","Data":"732e3b42123d9df0ef12b3a5b80dcfbbe3aa31ed9d92e875c3ec8984d0c825c6"}
Sep 14 10:10:36.817519 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:36.817339    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="732e3b42123d9df0ef12b3a5b80dcfbbe3aa31ed9d92e875c3ec8984d0c825c6"
Sep 14 10:10:37.258512 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:37.258478    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-03fd6d-openshift-default-5854564496-x76m9"
Sep 14 10:10:37.435954 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:37.435912    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3650d5df-8b7e-465f-83c2-65d3daecd3de-credential-socket\") pod \"3650d5df-8b7e-465f-83c2-65d3daecd3de\" (UID: \"3650d5df-8b7e-465f-83c2-65d3daecd3de\") "
Sep 14 10:10:37.435954 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:37.435955    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3650d5df-8b7e-465f-83c2-65d3daecd3de-istio-envoy\") pod \"3650d5df-8b7e-465f-83c2-65d3daecd3de\" (UID: \"3650d5df-8b7e-465f-83c2-65d3daecd3de\") "
Sep 14 10:10:37.436243 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:37.435991    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3650d5df-8b7e-465f-83c2-65d3daecd3de-istio-data\") pod \"3650d5df-8b7e-465f-83c2-65d3daecd3de\" (UID: \"3650d5df-8b7e-465f-83c2-65d3daecd3de\") "
Sep 14 10:10:37.436243 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:37.436038    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3650d5df-8b7e-465f-83c2-65d3daecd3de-workload-socket\") pod \"3650d5df-8b7e-465f-83c2-65d3daecd3de\" (UID: \"3650d5df-8b7e-465f-83c2-65d3daecd3de\") "
Sep 14 10:10:37.436243 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:37.436094    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3650d5df-8b7e-465f-83c2-65d3daecd3de-istiod-ca-cert\") pod \"3650d5df-8b7e-465f-83c2-65d3daecd3de\" (UID: \"3650d5df-8b7e-465f-83c2-65d3daecd3de\") "
Sep 14 10:10:37.436243 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:37.436155    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3650d5df-8b7e-465f-83c2-65d3daecd3de-istio-podinfo\") pod \"3650d5df-8b7e-465f-83c2-65d3daecd3de\" (UID: \"3650d5df-8b7e-465f-83c2-65d3daecd3de\") "
Sep 14 10:10:37.436243 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:37.436224    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3650d5df-8b7e-465f-83c2-65d3daecd3de-workload-certs\") pod \"3650d5df-8b7e-465f-83c2-65d3daecd3de\" (UID: \"3650d5df-8b7e-465f-83c2-65d3daecd3de\") "
Sep 14 10:10:37.436492 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:37.436255    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ljb4j\" (UniqueName: \"kubernetes.io/projected/3650d5df-8b7e-465f-83c2-65d3daecd3de-kube-api-access-ljb4j\") pod \"3650d5df-8b7e-465f-83c2-65d3daecd3de\" (UID: \"3650d5df-8b7e-465f-83c2-65d3daecd3de\") "
Sep 14 10:10:37.436492 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:37.436304    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3650d5df-8b7e-465f-83c2-65d3daecd3de-istio-token\") pod \"3650d5df-8b7e-465f-83c2-65d3daecd3de\" (UID: \"3650d5df-8b7e-465f-83c2-65d3daecd3de\") "
Sep 14 10:10:37.436492 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:37.436344    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3650d5df-8b7e-465f-83c2-65d3daecd3de-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3650d5df-8b7e-465f-83c2-65d3daecd3de" (UID: "3650d5df-8b7e-465f-83c2-65d3daecd3de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:10:37.436492 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:37.436355    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3650d5df-8b7e-465f-83c2-65d3daecd3de-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3650d5df-8b7e-465f-83c2-65d3daecd3de" (UID: "3650d5df-8b7e-465f-83c2-65d3daecd3de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:10:37.436492 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:37.436376    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3650d5df-8b7e-465f-83c2-65d3daecd3de-istio-data" (OuterVolumeSpecName: "istio-data") pod "3650d5df-8b7e-465f-83c2-65d3daecd3de" (UID: "3650d5df-8b7e-465f-83c2-65d3daecd3de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:10:37.436756 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:37.436559    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3650d5df-8b7e-465f-83c2-65d3daecd3de-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3650d5df-8b7e-465f-83c2-65d3daecd3de" (UID: "3650d5df-8b7e-465f-83c2-65d3daecd3de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:10:37.436756 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:37.436654    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3650d5df-8b7e-465f-83c2-65d3daecd3de-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3650d5df-8b7e-465f-83c2-65d3daecd3de" (UID: "3650d5df-8b7e-465f-83c2-65d3daecd3de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:10:37.436756 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:37.436673    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3650d5df-8b7e-465f-83c2-65d3daecd3de-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:10:37.436756 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:37.436690    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3650d5df-8b7e-465f-83c2-65d3daecd3de-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:10:37.436756 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:37.436700    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3650d5df-8b7e-465f-83c2-65d3daecd3de-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:10:37.438246 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:37.438217    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3650d5df-8b7e-465f-83c2-65d3daecd3de-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3650d5df-8b7e-465f-83c2-65d3daecd3de" (UID: "3650d5df-8b7e-465f-83c2-65d3daecd3de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:10:37.438453 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:37.438435    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3650d5df-8b7e-465f-83c2-65d3daecd3de-istio-token" (OuterVolumeSpecName: "istio-token") pod "3650d5df-8b7e-465f-83c2-65d3daecd3de" (UID: "3650d5df-8b7e-465f-83c2-65d3daecd3de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:10:37.438860 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:37.438840    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3650d5df-8b7e-465f-83c2-65d3daecd3de-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3650d5df-8b7e-465f-83c2-65d3daecd3de" (UID: "3650d5df-8b7e-465f-83c2-65d3daecd3de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 10:10:37.438919 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:37.438840    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3650d5df-8b7e-465f-83c2-65d3daecd3de-kube-api-access-ljb4j" (OuterVolumeSpecName: "kube-api-access-ljb4j") pod "3650d5df-8b7e-465f-83c2-65d3daecd3de" (UID: "3650d5df-8b7e-465f-83c2-65d3daecd3de"). InnerVolumeSpecName "kube-api-access-ljb4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:10:37.482627 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:37.482587    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:37.538250 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:37.538205    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3650d5df-8b7e-465f-83c2-65d3daecd3de-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:10:37.538250 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:37.538244    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3650d5df-8b7e-465f-83c2-65d3daecd3de-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:10:37.538250 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:37.538255    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3650d5df-8b7e-465f-83c2-65d3daecd3de-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:10:37.538494 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:37.538265    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljb4j\" (UniqueName: \"kubernetes.io/projected/3650d5df-8b7e-465f-83c2-65d3daecd3de-kube-api-access-ljb4j\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:10:37.538494 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:37.538276    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3650d5df-8b7e-465f-83c2-65d3daecd3de-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:10:37.538494 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:37.538284    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3650d5df-8b7e-465f-83c2-65d3daecd3de-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:10:37.826150 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:37.826057    2563 generic.go:356] "Generic (PLEG): container finished" podID="3650d5df-8b7e-465f-83c2-65d3daecd3de" containerID="17b1b6fa9a0bfb64c43ee439bbde2b713ef7a8fd3531a9a039925fcf4346dc86" exitCode=0
Sep 14 10:10:37.826150 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:37.826108    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-03fd6d-openshift-default-5854564496-x76m9"
Sep 14 10:10:37.826342 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:37.826143    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-03fd6d-openshift-default-5854564496-x76m9" event={"ID":"3650d5df-8b7e-465f-83c2-65d3daecd3de","Type":"ContainerDied","Data":"17b1b6fa9a0bfb64c43ee439bbde2b713ef7a8fd3531a9a039925fcf4346dc86"}
Sep 14 10:10:37.826342 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:37.826202    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-03fd6d-openshift-default-5854564496-x76m9" event={"ID":"3650d5df-8b7e-465f-83c2-65d3daecd3de","Type":"ContainerDied","Data":"aecd8f4d2169f0d08391956e528c1d3bf7889f1455066496ede8a9aac32e67b2"}
Sep 14 10:10:37.826342 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:37.826232    2563 scope.go:117] "RemoveContainer" containerID="17b1b6fa9a0bfb64c43ee439bbde2b713ef7a8fd3531a9a039925fcf4346dc86"
Sep 14 10:10:37.836883 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:37.836859    2563 scope.go:117] "RemoveContainer" containerID="0d4a420373f7d148687d32fe245e7f2b7076632a6592f2ce61a42e9dd8643767"
Sep 14 10:10:37.844104 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:37.844070    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-03fd6d-openshift-default-5854564496-x76m9"]
Sep 14 10:10:37.848036 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:37.847991    2563 scope.go:117] "RemoveContainer" containerID="17b1b6fa9a0bfb64c43ee439bbde2b713ef7a8fd3531a9a039925fcf4346dc86"
Sep 14 10:10:37.848478 ip-10-0-132-4 kubenswrapper[2563]: E0914 10:10:37.848451    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17b1b6fa9a0bfb64c43ee439bbde2b713ef7a8fd3531a9a039925fcf4346dc86\": container with ID starting with 17b1b6fa9a0bfb64c43ee439bbde2b713ef7a8fd3531a9a039925fcf4346dc86 not found: ID does not exist" containerID="17b1b6fa9a0bfb64c43ee439bbde2b713ef7a8fd3531a9a039925fcf4346dc86"
Sep 14 10:10:37.848592 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:37.848489    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17b1b6fa9a0bfb64c43ee439bbde2b713ef7a8fd3531a9a039925fcf4346dc86"} err="failed to get container status \"17b1b6fa9a0bfb64c43ee439bbde2b713ef7a8fd3531a9a039925fcf4346dc86\": rpc error: code = NotFound desc = could not find container \"17b1b6fa9a0bfb64c43ee439bbde2b713ef7a8fd3531a9a039925fcf4346dc86\": container with ID starting with 17b1b6fa9a0bfb64c43ee439bbde2b713ef7a8fd3531a9a039925fcf4346dc86 not found: ID does not exist"
Sep 14 10:10:37.848592 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:37.848522    2563 scope.go:117] "RemoveContainer" containerID="0d4a420373f7d148687d32fe245e7f2b7076632a6592f2ce61a42e9dd8643767"
Sep 14 10:10:37.848866 ip-10-0-132-4 kubenswrapper[2563]: E0914 10:10:37.848841    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d4a420373f7d148687d32fe245e7f2b7076632a6592f2ce61a42e9dd8643767\": container with ID starting with 0d4a420373f7d148687d32fe245e7f2b7076632a6592f2ce61a42e9dd8643767 not found: ID does not exist" containerID="0d4a420373f7d148687d32fe245e7f2b7076632a6592f2ce61a42e9dd8643767"
Sep 14 10:10:37.848919 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:37.848878    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d4a420373f7d148687d32fe245e7f2b7076632a6592f2ce61a42e9dd8643767"} err="failed to get container status \"0d4a420373f7d148687d32fe245e7f2b7076632a6592f2ce61a42e9dd8643767\": rpc error: code = NotFound desc = could not find container \"0d4a420373f7d148687d32fe245e7f2b7076632a6592f2ce61a42e9dd8643767\": container with ID starting with 0d4a420373f7d148687d32fe245e7f2b7076632a6592f2ce61a42e9dd8643767 not found: ID does not exist"
Sep 14 10:10:37.850531 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:37.850507    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-03fd6d-openshift-default-5854564496-x76m9"]
Sep 14 10:10:38.307434 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:38.307397    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:39.482677 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:39.482634    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3650d5df-8b7e-465f-83c2-65d3daecd3de" path="/var/lib/kubelet/pods/3650d5df-8b7e-465f-83c2-65d3daecd3de/volumes"
Sep 14 10:10:40.640619 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:40.640579    2563 patch_prober.go:28] interesting pod/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 14 10:10:40.641083 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:40.640644    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw" podUID="a9d150e6-194e-44be-8a5f-bb826032f664" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 14 10:10:41.535787 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:41.535700    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw"
Sep 14 10:10:41.573724 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:41.573687    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9d150e6-194e-44be-8a5f-bb826032f664-istio-podinfo\") pod \"a9d150e6-194e-44be-8a5f-bb826032f664\" (UID: \"a9d150e6-194e-44be-8a5f-bb826032f664\") "
Sep 14 10:10:41.573884 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:41.573733    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9d150e6-194e-44be-8a5f-bb826032f664-istio-token\") pod \"a9d150e6-194e-44be-8a5f-bb826032f664\" (UID: \"a9d150e6-194e-44be-8a5f-bb826032f664\") "
Sep 14 10:10:41.573884 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:41.573774    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9d150e6-194e-44be-8a5f-bb826032f664-istiod-ca-cert\") pod \"a9d150e6-194e-44be-8a5f-bb826032f664\" (UID: \"a9d150e6-194e-44be-8a5f-bb826032f664\") "
Sep 14 10:10:41.573884 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:41.573815    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9d150e6-194e-44be-8a5f-bb826032f664-workload-certs\") pod \"a9d150e6-194e-44be-8a5f-bb826032f664\" (UID: \"a9d150e6-194e-44be-8a5f-bb826032f664\") "
Sep 14 10:10:41.574047 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:41.573885    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9d150e6-194e-44be-8a5f-bb826032f664-workload-socket\") pod \"a9d150e6-194e-44be-8a5f-bb826032f664\" (UID: \"a9d150e6-194e-44be-8a5f-bb826032f664\") "
Sep 14 10:10:41.574047 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:41.573936    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9d150e6-194e-44be-8a5f-bb826032f664-istio-data\") pod \"a9d150e6-194e-44be-8a5f-bb826032f664\" (UID: \"a9d150e6-194e-44be-8a5f-bb826032f664\") "
Sep 14 10:10:41.574047 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:41.573964    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9d150e6-194e-44be-8a5f-bb826032f664-credential-socket\") pod \"a9d150e6-194e-44be-8a5f-bb826032f664\" (UID: \"a9d150e6-194e-44be-8a5f-bb826032f664\") "
Sep 14 10:10:41.574215 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:41.574046    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bvkbs\" (UniqueName: \"kubernetes.io/projected/a9d150e6-194e-44be-8a5f-bb826032f664-kube-api-access-bvkbs\") pod \"a9d150e6-194e-44be-8a5f-bb826032f664\" (UID: \"a9d150e6-194e-44be-8a5f-bb826032f664\") "
Sep 14 10:10:41.574215 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:41.574081    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9d150e6-194e-44be-8a5f-bb826032f664-istio-envoy\") pod \"a9d150e6-194e-44be-8a5f-bb826032f664\" (UID: \"a9d150e6-194e-44be-8a5f-bb826032f664\") "
Sep 14 10:10:41.574215 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:41.574153    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9d150e6-194e-44be-8a5f-bb826032f664-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a9d150e6-194e-44be-8a5f-bb826032f664" (UID: "a9d150e6-194e-44be-8a5f-bb826032f664"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:10:41.574367 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:41.574239    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9d150e6-194e-44be-8a5f-bb826032f664-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a9d150e6-194e-44be-8a5f-bb826032f664" (UID: "a9d150e6-194e-44be-8a5f-bb826032f664"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:10:41.574367 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:41.574257    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9d150e6-194e-44be-8a5f-bb826032f664-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a9d150e6-194e-44be-8a5f-bb826032f664" (UID: "a9d150e6-194e-44be-8a5f-bb826032f664"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:10:41.574367 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:41.574262    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9d150e6-194e-44be-8a5f-bb826032f664-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a9d150e6-194e-44be-8a5f-bb826032f664" (UID: "a9d150e6-194e-44be-8a5f-bb826032f664"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:10:41.574576 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:41.574556    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9d150e6-194e-44be-8a5f-bb826032f664-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:10:41.574643 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:41.574582    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9d150e6-194e-44be-8a5f-bb826032f664-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:10:41.574643 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:41.574596    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9d150e6-194e-44be-8a5f-bb826032f664-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:10:41.574643 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:41.574597    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9d150e6-194e-44be-8a5f-bb826032f664-istio-data" (OuterVolumeSpecName: "istio-data") pod "a9d150e6-194e-44be-8a5f-bb826032f664" (UID: "a9d150e6-194e-44be-8a5f-bb826032f664"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:10:41.574643 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:41.574608    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9d150e6-194e-44be-8a5f-bb826032f664-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:10:41.576724 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:41.576676    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9d150e6-194e-44be-8a5f-bb826032f664-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a9d150e6-194e-44be-8a5f-bb826032f664" (UID: "a9d150e6-194e-44be-8a5f-bb826032f664"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:10:41.576724 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:41.576712    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a9d150e6-194e-44be-8a5f-bb826032f664-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a9d150e6-194e-44be-8a5f-bb826032f664" (UID: "a9d150e6-194e-44be-8a5f-bb826032f664"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 10:10:41.576991 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:41.576848    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9d150e6-194e-44be-8a5f-bb826032f664-istio-token" (OuterVolumeSpecName: "istio-token") pod "a9d150e6-194e-44be-8a5f-bb826032f664" (UID: "a9d150e6-194e-44be-8a5f-bb826032f664"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:10:41.576991 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:41.576874    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9d150e6-194e-44be-8a5f-bb826032f664-kube-api-access-bvkbs" (OuterVolumeSpecName: "kube-api-access-bvkbs") pod "a9d150e6-194e-44be-8a5f-bb826032f664" (UID: "a9d150e6-194e-44be-8a5f-bb826032f664"). InnerVolumeSpecName "kube-api-access-bvkbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:10:41.675191 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:41.675147    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvkbs\" (UniqueName: \"kubernetes.io/projected/a9d150e6-194e-44be-8a5f-bb826032f664-kube-api-access-bvkbs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:10:41.675191 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:41.675183    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9d150e6-194e-44be-8a5f-bb826032f664-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:10:41.675191 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:41.675193    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9d150e6-194e-44be-8a5f-bb826032f664-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:10:41.675191 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:41.675201    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9d150e6-194e-44be-8a5f-bb826032f664-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:10:41.675191 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:41.675209    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9d150e6-194e-44be-8a5f-bb826032f664-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:10:41.855437 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:41.855339    2563 generic.go:356] "Generic (PLEG): container finished" podID="a9d150e6-194e-44be-8a5f-bb826032f664" containerID="606094d57bf0e58f8d128d7feda6dd7fe2c4919e29a3f387551845738a77ac18" exitCode=0
Sep 14 10:10:41.855437 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:41.855400    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw"
Sep 14 10:10:41.855643 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:41.855427    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw" event={"ID":"a9d150e6-194e-44be-8a5f-bb826032f664","Type":"ContainerDied","Data":"606094d57bf0e58f8d128d7feda6dd7fe2c4919e29a3f387551845738a77ac18"}
Sep 14 10:10:41.855643 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:41.855489    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw" event={"ID":"a9d150e6-194e-44be-8a5f-bb826032f664","Type":"ContainerDied","Data":"a8bce98008164e3c02bba4ecad6e48ab5d732a8ef7135f97c4b628876c276f12"}
Sep 14 10:10:41.855643 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:41.855512    2563 scope.go:117] "RemoveContainer" containerID="606094d57bf0e58f8d128d7feda6dd7fe2c4919e29a3f387551845738a77ac18"
Sep 14 10:10:41.866660 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:41.866631    2563 scope.go:117] "RemoveContainer" containerID="d42c378f47c1e6616509d23a6f306ae3a147e0c5e1bae984a4774ac1fe3aa63e"
Sep 14 10:10:41.876936 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:41.876903    2563 scope.go:117] "RemoveContainer" containerID="606094d57bf0e58f8d128d7feda6dd7fe2c4919e29a3f387551845738a77ac18"
Sep 14 10:10:41.877409 ip-10-0-132-4 kubenswrapper[2563]: E0914 10:10:41.877379    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"606094d57bf0e58f8d128d7feda6dd7fe2c4919e29a3f387551845738a77ac18\": container with ID starting with 606094d57bf0e58f8d128d7feda6dd7fe2c4919e29a3f387551845738a77ac18 not found: ID does not exist" containerID="606094d57bf0e58f8d128d7feda6dd7fe2c4919e29a3f387551845738a77ac18"
Sep 14 10:10:41.877516 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:41.877419    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"606094d57bf0e58f8d128d7feda6dd7fe2c4919e29a3f387551845738a77ac18"} err="failed to get container status \"606094d57bf0e58f8d128d7feda6dd7fe2c4919e29a3f387551845738a77ac18\": rpc error: code = NotFound desc = could not find container \"606094d57bf0e58f8d128d7feda6dd7fe2c4919e29a3f387551845738a77ac18\": container with ID starting with 606094d57bf0e58f8d128d7feda6dd7fe2c4919e29a3f387551845738a77ac18 not found: ID does not exist"
Sep 14 10:10:41.877516 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:41.877446    2563 scope.go:117] "RemoveContainer" containerID="d42c378f47c1e6616509d23a6f306ae3a147e0c5e1bae984a4774ac1fe3aa63e"
Sep 14 10:10:41.877773 ip-10-0-132-4 kubenswrapper[2563]: E0914 10:10:41.877748    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d42c378f47c1e6616509d23a6f306ae3a147e0c5e1bae984a4774ac1fe3aa63e\": container with ID starting with d42c378f47c1e6616509d23a6f306ae3a147e0c5e1bae984a4774ac1fe3aa63e not found: ID does not exist" containerID="d42c378f47c1e6616509d23a6f306ae3a147e0c5e1bae984a4774ac1fe3aa63e"
Sep 14 10:10:41.877822 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:41.877780    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d42c378f47c1e6616509d23a6f306ae3a147e0c5e1bae984a4774ac1fe3aa63e"} err="failed to get container status \"d42c378f47c1e6616509d23a6f306ae3a147e0c5e1bae984a4774ac1fe3aa63e\": rpc error: code = NotFound desc = could not find container \"d42c378f47c1e6616509d23a6f306ae3a147e0c5e1bae984a4774ac1fe3aa63e\": container with ID starting with d42c378f47c1e6616509d23a6f306ae3a147e0c5e1bae984a4774ac1fe3aa63e not found: ID does not exist"
Sep 14 10:10:41.880329 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:41.880288    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw"]
Sep 14 10:10:41.883134 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:41.883100    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b6812e-openshift-default-68886d656-tr7hw"]
Sep 14 10:10:42.814855 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:42.814801    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:43.485493 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:43.485453    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9d150e6-194e-44be-8a5f-bb826032f664" path="/var/lib/kubelet/pods/a9d150e6-194e-44be-8a5f-bb826032f664/volumes"
Sep 14 10:10:43.500949 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:43.500914    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:43.598186 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:43.598151    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:43.672149 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:43.672115    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:43.758867 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:43.758760    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8fd006-ldjrt"]
Sep 14 10:10:43.761487 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:43.761456    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:43.762637 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:43.762612    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8fd006-ldjrt"]
Sep 14 10:10:43.857154 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:43.857115    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:43.994584 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:43.994542    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:44.002822 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:44.002785    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2901ec-29823000-mn5kc"]
Sep 14 10:10:44.007189 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:44.007148    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2901ec-29823000-mn5kc"]
Sep 14 10:10:44.085139 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:44.085056    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:44.152348 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:44.152307    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:44.244022 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:44.243978    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:44.319674 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:44.319640    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:44.337408 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:44.337305    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2901ec-9krtq"]
Sep 14 10:10:44.348730 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:44.348690    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2901ec-9krtq"]
Sep 14 10:10:44.655625 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:44.655529    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:44.769758 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:44.769726    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:45.473259 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:45.473223    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:45.482986 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:45.482950    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05627ca1-2838-4242-bad6-069eae834fc9" path="/var/lib/kubelet/pods/05627ca1-2838-4242-bad6-069eae834fc9/volumes"
Sep 14 10:10:45.483320 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:45.483305    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="873f88a1-9881-41e8-8bc2-cff73ebbfed7" path="/var/lib/kubelet/pods/873f88a1-9881-41e8-8bc2-cff73ebbfed7/volumes"
Sep 14 10:10:45.483597 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:45.483584    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8c2b453-d296-45c9-a279-0ff272cc3212" path="/var/lib/kubelet/pods/f8c2b453-d296-45c9-a279-0ff272cc3212/volumes"
Sep 14 10:10:46.258214 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:46.258175    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:47.003737 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:47.003694    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:47.374529 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:47.374432    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-2901ec/distinct-w4-2901ec-kserve-9fc4b5697-xblp8" podUID="3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc" containerName="main" containerID="cri-o://f398d11a46bcde229405a18228763a69badb0d7b2bf6da64c7ef895d363c3ec9" gracePeriod=45
Sep 14 10:10:47.425638 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:47.425591    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-2901ec/simulator-w4-2901ec-kserve-7c4cdb8dff-nhllq" podUID="0d8c1e42-d4d8-456c-9cc1-1a493e8e3835" containerName="main" containerID="cri-o://3d2010bbad19cdef1c49b5bb819462c6d099207704c164d0ea3d76f8bd795e96" gracePeriod=45
Sep 14 10:10:47.728340 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:47.728302    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2901ec/distinct-w4-2901ec-kserve-9fc4b5697-xblp8"
Sep 14 10:10:47.733263 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:47.733229    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2901ec/simulator-w4-2901ec-kserve-7c4cdb8dff-nhllq"
Sep 14 10:10:47.823749 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:47.823705    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc-dshm\") pod \"3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc\" (UID: \"3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc\") "
Sep 14 10:10:47.823951 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:47.823784    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc-tls-certs\") pod \"3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc\" (UID: \"3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc\") "
Sep 14 10:10:47.823951 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:47.823837    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d8c1e42-d4d8-456c-9cc1-1a493e8e3835-home\") pod \"0d8c1e42-d4d8-456c-9cc1-1a493e8e3835\" (UID: \"0d8c1e42-d4d8-456c-9cc1-1a493e8e3835\") "
Sep 14 10:10:47.823951 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:47.823876    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d8c1e42-d4d8-456c-9cc1-1a493e8e3835-tls-certs\") pod \"0d8c1e42-d4d8-456c-9cc1-1a493e8e3835\" (UID: \"0d8c1e42-d4d8-456c-9cc1-1a493e8e3835\") "
Sep 14 10:10:47.823951 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:47.823908    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vfk6\" (UniqueName: \"kubernetes.io/projected/3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc-kube-api-access-4vfk6\") pod \"3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc\" (UID: \"3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc\") "
Sep 14 10:10:47.823951 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:47.823941    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d8c1e42-d4d8-456c-9cc1-1a493e8e3835-dshm\") pod \"0d8c1e42-d4d8-456c-9cc1-1a493e8e3835\" (UID: \"0d8c1e42-d4d8-456c-9cc1-1a493e8e3835\") "
Sep 14 10:10:47.824269 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:47.823965    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc-tmp-dir\") pod \"3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc\" (UID: \"3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc\") "
Sep 14 10:10:47.824269 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:47.824068    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d8c1e42-d4d8-456c-9cc1-1a493e8e3835-tmp-dir\") pod \"0d8c1e42-d4d8-456c-9cc1-1a493e8e3835\" (UID: \"0d8c1e42-d4d8-456c-9cc1-1a493e8e3835\") "
Sep 14 10:10:47.824269 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:47.824092    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc-home\") pod \"3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc\" (UID: \"3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc\") "
Sep 14 10:10:47.824269 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:47.824132    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc-model-cache\") pod \"3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc\" (UID: \"3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc\") "
Sep 14 10:10:47.824269 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:47.824161    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d8c1e42-d4d8-456c-9cc1-1a493e8e3835-model-cache\") pod \"0d8c1e42-d4d8-456c-9cc1-1a493e8e3835\" (UID: \"0d8c1e42-d4d8-456c-9cc1-1a493e8e3835\") "
Sep 14 10:10:47.824269 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:47.824159    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d8c1e42-d4d8-456c-9cc1-1a493e8e3835-home" (OuterVolumeSpecName: "home") pod "0d8c1e42-d4d8-456c-9cc1-1a493e8e3835" (UID: "0d8c1e42-d4d8-456c-9cc1-1a493e8e3835"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:10:47.824269 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:47.824198    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c2q2x\" (UniqueName: \"kubernetes.io/projected/0d8c1e42-d4d8-456c-9cc1-1a493e8e3835-kube-api-access-c2q2x\") pod \"0d8c1e42-d4d8-456c-9cc1-1a493e8e3835\" (UID: \"0d8c1e42-d4d8-456c-9cc1-1a493e8e3835\") "
Sep 14 10:10:47.824653 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:47.824373    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc" (UID: "3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:10:47.824653 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:47.824383    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc-home" (OuterVolumeSpecName: "home") pod "3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc" (UID: "3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:10:47.824653 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:47.824579    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d8c1e42-d4d8-456c-9cc1-1a493e8e3835-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0d8c1e42-d4d8-456c-9cc1-1a493e8e3835" (UID: "0d8c1e42-d4d8-456c-9cc1-1a493e8e3835"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:10:47.824822 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:47.824670    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d8c1e42-d4d8-456c-9cc1-1a493e8e3835-home\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:10:47.824822 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:47.824711    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc-tmp-dir\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:10:47.824822 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:47.824728    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d8c1e42-d4d8-456c-9cc1-1a493e8e3835-tmp-dir\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:10:47.824822 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:47.824763    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc-home\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:10:47.824822 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:47.824803    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc-model-cache" (OuterVolumeSpecName: "model-cache") pod "3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc" (UID: "3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:10:47.825115 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:47.824949    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d8c1e42-d4d8-456c-9cc1-1a493e8e3835-model-cache" (OuterVolumeSpecName: "model-cache") pod "0d8c1e42-d4d8-456c-9cc1-1a493e8e3835" (UID: "0d8c1e42-d4d8-456c-9cc1-1a493e8e3835"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:10:47.826759 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:47.826721    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc" (UID: "3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:10:47.826759 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:47.826737    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc-kube-api-access-4vfk6" (OuterVolumeSpecName: "kube-api-access-4vfk6") pod "3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc" (UID: "3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc"). InnerVolumeSpecName "kube-api-access-4vfk6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:10:47.826962 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:47.826914    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d8c1e42-d4d8-456c-9cc1-1a493e8e3835-kube-api-access-c2q2x" (OuterVolumeSpecName: "kube-api-access-c2q2x") pod "0d8c1e42-d4d8-456c-9cc1-1a493e8e3835" (UID: "0d8c1e42-d4d8-456c-9cc1-1a493e8e3835"). InnerVolumeSpecName "kube-api-access-c2q2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:10:47.827080 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:47.827054    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d8c1e42-d4d8-456c-9cc1-1a493e8e3835-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0d8c1e42-d4d8-456c-9cc1-1a493e8e3835" (UID: "0d8c1e42-d4d8-456c-9cc1-1a493e8e3835"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:10:47.827080 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:47.827069    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc-dshm" (OuterVolumeSpecName: "dshm") pod "3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc" (UID: "3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:10:47.827175 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:47.827079    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d8c1e42-d4d8-456c-9cc1-1a493e8e3835-dshm" (OuterVolumeSpecName: "dshm") pod "0d8c1e42-d4d8-456c-9cc1-1a493e8e3835" (UID: "0d8c1e42-d4d8-456c-9cc1-1a493e8e3835"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:10:47.907534 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:47.907426    2563 generic.go:356] "Generic (PLEG): container finished" podID="0d8c1e42-d4d8-456c-9cc1-1a493e8e3835" containerID="3d2010bbad19cdef1c49b5bb819462c6d099207704c164d0ea3d76f8bd795e96" exitCode=0
Sep 14 10:10:47.907534 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:47.907476    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2901ec/simulator-w4-2901ec-kserve-7c4cdb8dff-nhllq"
Sep 14 10:10:47.907534 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:47.907517    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2901ec/simulator-w4-2901ec-kserve-7c4cdb8dff-nhllq" event={"ID":"0d8c1e42-d4d8-456c-9cc1-1a493e8e3835","Type":"ContainerDied","Data":"3d2010bbad19cdef1c49b5bb819462c6d099207704c164d0ea3d76f8bd795e96"}
Sep 14 10:10:47.907819 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:47.907565    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2901ec/simulator-w4-2901ec-kserve-7c4cdb8dff-nhllq" event={"ID":"0d8c1e42-d4d8-456c-9cc1-1a493e8e3835","Type":"ContainerDied","Data":"1dd5b2106fc400efa45b6bbe87a10d08518f47e8a064c678ddb7bd5e7f743233"}
Sep 14 10:10:47.907819 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:47.907587    2563 scope.go:117] "RemoveContainer" containerID="3d2010bbad19cdef1c49b5bb819462c6d099207704c164d0ea3d76f8bd795e96"
Sep 14 10:10:47.909106 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:47.909080    2563 generic.go:356] "Generic (PLEG): container finished" podID="3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc" containerID="f398d11a46bcde229405a18228763a69badb0d7b2bf6da64c7ef895d363c3ec9" exitCode=0
Sep 14 10:10:47.909231 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:47.909111    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2901ec/distinct-w4-2901ec-kserve-9fc4b5697-xblp8" event={"ID":"3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc","Type":"ContainerDied","Data":"f398d11a46bcde229405a18228763a69badb0d7b2bf6da64c7ef895d363c3ec9"}
Sep 14 10:10:47.909231 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:47.909140    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2901ec/distinct-w4-2901ec-kserve-9fc4b5697-xblp8"
Sep 14 10:10:47.909231 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:47.909160    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2901ec/distinct-w4-2901ec-kserve-9fc4b5697-xblp8" event={"ID":"3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc","Type":"ContainerDied","Data":"a7ddfd69422e2f21fd7cacce95aca3083ac0ee91cf93d791c7cc6bdc6bfc613e"}
Sep 14 10:10:47.918206 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:47.918181    2563 scope.go:117] "RemoveContainer" containerID="3d2010bbad19cdef1c49b5bb819462c6d099207704c164d0ea3d76f8bd795e96"
Sep 14 10:10:47.918663 ip-10-0-132-4 kubenswrapper[2563]: E0914 10:10:47.918639    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d2010bbad19cdef1c49b5bb819462c6d099207704c164d0ea3d76f8bd795e96\": container with ID starting with 3d2010bbad19cdef1c49b5bb819462c6d099207704c164d0ea3d76f8bd795e96 not found: ID does not exist" containerID="3d2010bbad19cdef1c49b5bb819462c6d099207704c164d0ea3d76f8bd795e96"
Sep 14 10:10:47.918781 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:47.918675    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d2010bbad19cdef1c49b5bb819462c6d099207704c164d0ea3d76f8bd795e96"} err="failed to get container status \"3d2010bbad19cdef1c49b5bb819462c6d099207704c164d0ea3d76f8bd795e96\": rpc error: code = NotFound desc = could not find container \"3d2010bbad19cdef1c49b5bb819462c6d099207704c164d0ea3d76f8bd795e96\": container with ID starting with 3d2010bbad19cdef1c49b5bb819462c6d099207704c164d0ea3d76f8bd795e96 not found: ID does not exist"
Sep 14 10:10:47.918781 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:47.918701    2563 scope.go:117] "RemoveContainer" containerID="f398d11a46bcde229405a18228763a69badb0d7b2bf6da64c7ef895d363c3ec9"
Sep 14 10:10:47.925625 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:47.925597    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d8c1e42-d4d8-456c-9cc1-1a493e8e3835-tls-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:10:47.925625 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:47.925621    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vfk6\" (UniqueName: \"kubernetes.io/projected/3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc-kube-api-access-4vfk6\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:10:47.925625 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:47.925631    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d8c1e42-d4d8-456c-9cc1-1a493e8e3835-dshm\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:10:47.925842 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:47.925641    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc-model-cache\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:10:47.925842 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:47.925650    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d8c1e42-d4d8-456c-9cc1-1a493e8e3835-model-cache\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:10:47.925842 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:47.925658    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2q2x\" (UniqueName: \"kubernetes.io/projected/0d8c1e42-d4d8-456c-9cc1-1a493e8e3835-kube-api-access-c2q2x\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:10:47.925842 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:47.925666    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc-dshm\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:10:47.925842 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:47.925673    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc-tls-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:10:47.928417 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:47.928392    2563 scope.go:117] "RemoveContainer" containerID="f398d11a46bcde229405a18228763a69badb0d7b2bf6da64c7ef895d363c3ec9"
Sep 14 10:10:47.928779 ip-10-0-132-4 kubenswrapper[2563]: E0914 10:10:47.928755    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f398d11a46bcde229405a18228763a69badb0d7b2bf6da64c7ef895d363c3ec9\": container with ID starting with f398d11a46bcde229405a18228763a69badb0d7b2bf6da64c7ef895d363c3ec9 not found: ID does not exist" containerID="f398d11a46bcde229405a18228763a69badb0d7b2bf6da64c7ef895d363c3ec9"
Sep 14 10:10:47.928856 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:47.928793    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f398d11a46bcde229405a18228763a69badb0d7b2bf6da64c7ef895d363c3ec9"} err="failed to get container status \"f398d11a46bcde229405a18228763a69badb0d7b2bf6da64c7ef895d363c3ec9\": rpc error: code = NotFound desc = could not find container \"f398d11a46bcde229405a18228763a69badb0d7b2bf6da64c7ef895d363c3ec9\": container with ID starting with f398d11a46bcde229405a18228763a69badb0d7b2bf6da64c7ef895d363c3ec9 not found: ID does not exist"
Sep 14 10:10:47.936153 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:47.936119    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-2901ec/distinct-w4-2901ec-kserve-9fc4b5697-xblp8"]
Sep 14 10:10:47.939755 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:47.939720    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-2901ec/distinct-w4-2901ec-kserve-9fc4b5697-xblp8"]
Sep 14 10:10:47.955624 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:47.955581    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-2901ec/simulator-w4-2901ec-kserve-7c4cdb8dff-nhllq"]
Sep 14 10:10:47.963131 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:47.963095    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-2901ec/simulator-w4-2901ec-kserve-7c4cdb8dff-nhllq"]
Sep 14 10:10:48.002109 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:48.001994    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:49.482082 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:49.482045    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d8c1e42-d4d8-456c-9cc1-1a493e8e3835" path="/var/lib/kubelet/pods/0d8c1e42-d4d8-456c-9cc1-1a493e8e3835/volumes"
Sep 14 10:10:49.482508 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:49.482494    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc" path="/var/lib/kubelet/pods/3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc/volumes"
Sep 14 10:10:50.183261 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:50.183228    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:50.445833 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:50.445734    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:50.518850 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:50.518800    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:50.694552 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:50.694514    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:51.008890 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:51.008845    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9e19b7-xjf6m"]
Sep 14 10:10:51.009270 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:51.009256    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9d150e6-194e-44be-8a5f-bb826032f664" containerName="istio-proxy"
Sep 14 10:10:51.009270 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:51.009271    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9d150e6-194e-44be-8a5f-bb826032f664" containerName="istio-proxy"
Sep 14 10:10:51.009366 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:51.009284    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9d150e6-194e-44be-8a5f-bb826032f664" containerName="istio-proxy"
Sep 14 10:10:51.009366 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:51.009289    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9d150e6-194e-44be-8a5f-bb826032f664" containerName="istio-proxy"
Sep 14 10:10:51.009366 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:51.009299    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc" containerName="main"
Sep 14 10:10:51.009366 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:51.009305    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc" containerName="main"
Sep 14 10:10:51.009366 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:51.009312    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3650d5df-8b7e-465f-83c2-65d3daecd3de" containerName="istio-proxy"
Sep 14 10:10:51.009366 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:51.009318    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3650d5df-8b7e-465f-83c2-65d3daecd3de" containerName="istio-proxy"
Sep 14 10:10:51.009366 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:51.009326    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3650d5df-8b7e-465f-83c2-65d3daecd3de" containerName="istio-proxy"
Sep 14 10:10:51.009366 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:51.009331    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3650d5df-8b7e-465f-83c2-65d3daecd3de" containerName="istio-proxy"
Sep 14 10:10:51.009366 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:51.009351    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d8c1e42-d4d8-456c-9cc1-1a493e8e3835" containerName="main"
Sep 14 10:10:51.009366 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:51.009356    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d8c1e42-d4d8-456c-9cc1-1a493e8e3835" containerName="main"
Sep 14 10:10:51.009366 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:51.009363    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05627ca1-2838-4242-bad6-069eae834fc9" containerName="revoke-keys"
Sep 14 10:10:51.009366 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:51.009369    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="05627ca1-2838-4242-bad6-069eae834fc9" containerName="revoke-keys"
Sep 14 10:10:51.009722 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:51.009446    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9d150e6-194e-44be-8a5f-bb826032f664" containerName="istio-proxy"
Sep 14 10:10:51.009722 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:51.009456    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3650d5df-8b7e-465f-83c2-65d3daecd3de" containerName="istio-proxy"
Sep 14 10:10:51.009722 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:51.009463    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9d150e6-194e-44be-8a5f-bb826032f664" containerName="istio-proxy"
Sep 14 10:10:51.009722 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:51.009469    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3650d5df-8b7e-465f-83c2-65d3daecd3de" containerName="istio-proxy"
Sep 14 10:10:51.009722 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:51.009474    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d8c1e42-d4d8-456c-9cc1-1a493e8e3835" containerName="main"
Sep 14 10:10:51.009722 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:51.009481    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c15cf83-c6b9-4371-8e1e-3d5c3f6d1fbc" containerName="main"
Sep 14 10:10:51.009722 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:51.009487    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="05627ca1-2838-4242-bad6-069eae834fc9" containerName="revoke-keys"
Sep 14 10:10:51.014245 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:51.014219    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9e19b7-xjf6m"
Sep 14 10:10:51.016639 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:51.016603    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 14 10:10:51.017135 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:51.017062    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gf4xj\""
Sep 14 10:10:51.020862 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:51.020822    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9e19b7-xjf6m"]
Sep 14 10:10:51.060517 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:51.060471    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba8e78fa-5ad0-4ea3-b522-4d49eb6396d5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-9e19b7-xjf6m\" (UID: \"ba8e78fa-5ad0-4ea3-b522-4d49eb6396d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9e19b7-xjf6m"
Sep 14 10:10:51.161395 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:51.161333    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba8e78fa-5ad0-4ea3-b522-4d49eb6396d5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-9e19b7-xjf6m\" (UID: \"ba8e78fa-5ad0-4ea3-b522-4d49eb6396d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9e19b7-xjf6m"
Sep 14 10:10:51.162150 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:51.162125    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba8e78fa-5ad0-4ea3-b522-4d49eb6396d5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-9e19b7-xjf6m\" (UID: \"ba8e78fa-5ad0-4ea3-b522-4d49eb6396d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9e19b7-xjf6m"
Sep 14 10:10:51.327169 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:51.327066    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9e19b7-xjf6m"
Sep 14 10:10:51.348918 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:51.348882    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:51.605495 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:51.605463    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9e19b7-xjf6m"]
Sep 14 10:10:51.607506 ip-10-0-132-4 kubenswrapper[2563]: W0914 10:10:51.607464    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba8e78fa_5ad0_4ea3_b522_4d49eb6396d5.slice/crio-690b56c1857b815beb03aa4f0c99a1d656173f8c25a8f15562142fa3f9b18aa5 WatchSource:0}: Error finding container 690b56c1857b815beb03aa4f0c99a1d656173f8c25a8f15562142fa3f9b18aa5: Status 404 returned error can't find the container with id 690b56c1857b815beb03aa4f0c99a1d656173f8c25a8f15562142fa3f9b18aa5
Sep 14 10:10:51.939873 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:51.939829    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9e19b7-xjf6m" event={"ID":"ba8e78fa-5ad0-4ea3-b522-4d49eb6396d5","Type":"ContainerStarted","Data":"690b56c1857b815beb03aa4f0c99a1d656173f8c25a8f15562142fa3f9b18aa5"}
Sep 14 10:10:52.053760 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:52.053720    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:52.159704 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:52.159670    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:52.308960 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:52.308912    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m"]
Sep 14 10:10:52.309324 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:52.309295    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m" podUID="3188c970-5b86-4c03-95cc-12480aa11882" containerName="istio-proxy" containerID="cri-o://12e1ac8bfb0d6781d8fe8a63d1df308662a52a39990c18e86f122862fb25fb85" gracePeriod=30
Sep 14 10:10:52.315743 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:52.315704    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:52.416693 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:52.416657    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:52.611367 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:52.611276    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:52.948529 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:52.948433    2563 generic.go:356] "Generic (PLEG): container finished" podID="ba8e78fa-5ad0-4ea3-b522-4d49eb6396d5" containerID="75bf44e4176adacd7e580807eda54fdcf8489562089a25e62245acc4c8a9022a" exitCode=0
Sep 14 10:10:52.948691 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:52.948523    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9e19b7-xjf6m" event={"ID":"ba8e78fa-5ad0-4ea3-b522-4d49eb6396d5","Type":"ContainerDied","Data":"75bf44e4176adacd7e580807eda54fdcf8489562089a25e62245acc4c8a9022a"}
Sep 14 10:10:53.202863 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:53.202739    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:54.100967 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:54.100934    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9e19b7-xjf6m"
Sep 14 10:10:54.193821 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:54.193782    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba8e78fa-5ad0-4ea3-b522-4d49eb6396d5-maas-api-service-ca\") pod \"ba8e78fa-5ad0-4ea3-b522-4d49eb6396d5\" (UID: \"ba8e78fa-5ad0-4ea3-b522-4d49eb6396d5\") "
Sep 14 10:10:54.194180 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:54.194157    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba8e78fa-5ad0-4ea3-b522-4d49eb6396d5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ba8e78fa-5ad0-4ea3-b522-4d49eb6396d5" (UID: "ba8e78fa-5ad0-4ea3-b522-4d49eb6396d5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:10:54.286957 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:54.286892    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:54.295162 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:54.295122    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba8e78fa-5ad0-4ea3-b522-4d49eb6396d5-maas-api-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:10:54.962523 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:54.962492    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9e19b7-xjf6m"
Sep 14 10:10:54.962722 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:54.962532    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9e19b7-xjf6m" event={"ID":"ba8e78fa-5ad0-4ea3-b522-4d49eb6396d5","Type":"ContainerDied","Data":"690b56c1857b815beb03aa4f0c99a1d656173f8c25a8f15562142fa3f9b18aa5"}
Sep 14 10:10:54.962722 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:54.962586    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="690b56c1857b815beb03aa4f0c99a1d656173f8c25a8f15562142fa3f9b18aa5"
Sep 14 10:10:55.074389 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:55.074355    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:55.468050 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:55.467988    2563 patch_prober.go:28] interesting pod/e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 14 10:10:55.468469 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:55.468088    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m" podUID="3188c970-5b86-4c03-95cc-12480aa11882" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 14 10:10:57.571864 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:57.571828    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m"
Sep 14 10:10:57.631432 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:57.631339    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3188c970-5b86-4c03-95cc-12480aa11882-istio-token\") pod \"3188c970-5b86-4c03-95cc-12480aa11882\" (UID: \"3188c970-5b86-4c03-95cc-12480aa11882\") "
Sep 14 10:10:57.631432 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:57.631381    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3188c970-5b86-4c03-95cc-12480aa11882-workload-socket\") pod \"3188c970-5b86-4c03-95cc-12480aa11882\" (UID: \"3188c970-5b86-4c03-95cc-12480aa11882\") "
Sep 14 10:10:57.631432 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:57.631427    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3188c970-5b86-4c03-95cc-12480aa11882-istio-envoy\") pod \"3188c970-5b86-4c03-95cc-12480aa11882\" (UID: \"3188c970-5b86-4c03-95cc-12480aa11882\") "
Sep 14 10:10:57.631686 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:57.631442    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3188c970-5b86-4c03-95cc-12480aa11882-credential-socket\") pod \"3188c970-5b86-4c03-95cc-12480aa11882\" (UID: \"3188c970-5b86-4c03-95cc-12480aa11882\") "
Sep 14 10:10:57.631686 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:57.631503    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3188c970-5b86-4c03-95cc-12480aa11882-istio-data\") pod \"3188c970-5b86-4c03-95cc-12480aa11882\" (UID: \"3188c970-5b86-4c03-95cc-12480aa11882\") "
Sep 14 10:10:57.631686 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:57.631557    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3188c970-5b86-4c03-95cc-12480aa11882-workload-certs\") pod \"3188c970-5b86-4c03-95cc-12480aa11882\" (UID: \"3188c970-5b86-4c03-95cc-12480aa11882\") "
Sep 14 10:10:57.631686 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:57.631614    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3188c970-5b86-4c03-95cc-12480aa11882-istiod-ca-cert\") pod \"3188c970-5b86-4c03-95cc-12480aa11882\" (UID: \"3188c970-5b86-4c03-95cc-12480aa11882\") "
Sep 14 10:10:57.631686 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:57.631646    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3188c970-5b86-4c03-95cc-12480aa11882-istio-podinfo\") pod \"3188c970-5b86-4c03-95cc-12480aa11882\" (UID: \"3188c970-5b86-4c03-95cc-12480aa11882\") "
Sep 14 10:10:57.631686 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:57.631677    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mdp8m\" (UniqueName: \"kubernetes.io/projected/3188c970-5b86-4c03-95cc-12480aa11882-kube-api-access-mdp8m\") pod \"3188c970-5b86-4c03-95cc-12480aa11882\" (UID: \"3188c970-5b86-4c03-95cc-12480aa11882\") "
Sep 14 10:10:57.631992 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:57.631793    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3188c970-5b86-4c03-95cc-12480aa11882-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3188c970-5b86-4c03-95cc-12480aa11882" (UID: "3188c970-5b86-4c03-95cc-12480aa11882"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:10:57.631992 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:57.631809    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3188c970-5b86-4c03-95cc-12480aa11882-istio-data" (OuterVolumeSpecName: "istio-data") pod "3188c970-5b86-4c03-95cc-12480aa11882" (UID: "3188c970-5b86-4c03-95cc-12480aa11882"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:10:57.631992 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:57.631820    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3188c970-5b86-4c03-95cc-12480aa11882-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3188c970-5b86-4c03-95cc-12480aa11882" (UID: "3188c970-5b86-4c03-95cc-12480aa11882"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:10:57.631992 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:57.631968    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3188c970-5b86-4c03-95cc-12480aa11882-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3188c970-5b86-4c03-95cc-12480aa11882" (UID: "3188c970-5b86-4c03-95cc-12480aa11882"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:10:57.632251 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:57.632219    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3188c970-5b86-4c03-95cc-12480aa11882-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:10:57.632251 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:57.632235    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3188c970-5b86-4c03-95cc-12480aa11882-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3188c970-5b86-4c03-95cc-12480aa11882" (UID: "3188c970-5b86-4c03-95cc-12480aa11882"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:10:57.632342 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:57.632245    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3188c970-5b86-4c03-95cc-12480aa11882-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:10:57.632342 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:57.632284    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3188c970-5b86-4c03-95cc-12480aa11882-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:10:57.632342 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:57.632302    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3188c970-5b86-4c03-95cc-12480aa11882-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:10:57.633771 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:57.633744    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3188c970-5b86-4c03-95cc-12480aa11882-istio-token" (OuterVolumeSpecName: "istio-token") pod "3188c970-5b86-4c03-95cc-12480aa11882" (UID: "3188c970-5b86-4c03-95cc-12480aa11882"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:10:57.633891 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:57.633848    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3188c970-5b86-4c03-95cc-12480aa11882-kube-api-access-mdp8m" (OuterVolumeSpecName: "kube-api-access-mdp8m") pod "3188c970-5b86-4c03-95cc-12480aa11882" (UID: "3188c970-5b86-4c03-95cc-12480aa11882"). InnerVolumeSpecName "kube-api-access-mdp8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:10:57.634069 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:57.634053    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3188c970-5b86-4c03-95cc-12480aa11882-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3188c970-5b86-4c03-95cc-12480aa11882" (UID: "3188c970-5b86-4c03-95cc-12480aa11882"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:10:57.634132 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:57.634077    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3188c970-5b86-4c03-95cc-12480aa11882-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3188c970-5b86-4c03-95cc-12480aa11882" (UID: "3188c970-5b86-4c03-95cc-12480aa11882"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 10:10:57.733626 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:57.733574    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3188c970-5b86-4c03-95cc-12480aa11882-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:10:57.733626 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:57.733612    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3188c970-5b86-4c03-95cc-12480aa11882-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:10:57.733626 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:57.733622    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdp8m\" (UniqueName: \"kubernetes.io/projected/3188c970-5b86-4c03-95cc-12480aa11882-kube-api-access-mdp8m\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:10:57.733626 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:57.733632    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3188c970-5b86-4c03-95cc-12480aa11882-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:10:57.733626 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:57.733641    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3188c970-5b86-4c03-95cc-12480aa11882-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:10:57.985444 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:57.985405    2563 generic.go:356] "Generic (PLEG): container finished" podID="3188c970-5b86-4c03-95cc-12480aa11882" containerID="12e1ac8bfb0d6781d8fe8a63d1df308662a52a39990c18e86f122862fb25fb85" exitCode=0
Sep 14 10:10:57.985619 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:57.985460    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m"
Sep 14 10:10:57.985619 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:57.985482    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m" event={"ID":"3188c970-5b86-4c03-95cc-12480aa11882","Type":"ContainerDied","Data":"12e1ac8bfb0d6781d8fe8a63d1df308662a52a39990c18e86f122862fb25fb85"}
Sep 14 10:10:57.985619 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:57.985539    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m" event={"ID":"3188c970-5b86-4c03-95cc-12480aa11882","Type":"ContainerDied","Data":"757dcf3abd95c2a95f1635b44d20ab000e7d8a1e8cf5294f69c0a0e7c1dcc234"}
Sep 14 10:10:57.985619 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:57.985565    2563 scope.go:117] "RemoveContainer" containerID="12e1ac8bfb0d6781d8fe8a63d1df308662a52a39990c18e86f122862fb25fb85"
Sep 14 10:10:57.996241 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:57.996220    2563 scope.go:117] "RemoveContainer" containerID="f7e0b1923c1515b6523444260d8e8f86a71e1c4c8e8a06270c6cbc3fa374ab22"
Sep 14 10:10:58.010187 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:58.010154    2563 scope.go:117] "RemoveContainer" containerID="12e1ac8bfb0d6781d8fe8a63d1df308662a52a39990c18e86f122862fb25fb85"
Sep 14 10:10:58.010369 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:58.010318    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m"]
Sep 14 10:10:58.010719 ip-10-0-132-4 kubenswrapper[2563]: E0914 10:10:58.010694    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12e1ac8bfb0d6781d8fe8a63d1df308662a52a39990c18e86f122862fb25fb85\": container with ID starting with 12e1ac8bfb0d6781d8fe8a63d1df308662a52a39990c18e86f122862fb25fb85 not found: ID does not exist" containerID="12e1ac8bfb0d6781d8fe8a63d1df308662a52a39990c18e86f122862fb25fb85"
Sep 14 10:10:58.010804 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:58.010732    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12e1ac8bfb0d6781d8fe8a63d1df308662a52a39990c18e86f122862fb25fb85"} err="failed to get container status \"12e1ac8bfb0d6781d8fe8a63d1df308662a52a39990c18e86f122862fb25fb85\": rpc error: code = NotFound desc = could not find container \"12e1ac8bfb0d6781d8fe8a63d1df308662a52a39990c18e86f122862fb25fb85\": container with ID starting with 12e1ac8bfb0d6781d8fe8a63d1df308662a52a39990c18e86f122862fb25fb85 not found: ID does not exist"
Sep 14 10:10:58.010804 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:58.010752    2563 scope.go:117] "RemoveContainer" containerID="f7e0b1923c1515b6523444260d8e8f86a71e1c4c8e8a06270c6cbc3fa374ab22"
Sep 14 10:10:58.011104 ip-10-0-132-4 kubenswrapper[2563]: E0914 10:10:58.011083    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7e0b1923c1515b6523444260d8e8f86a71e1c4c8e8a06270c6cbc3fa374ab22\": container with ID starting with f7e0b1923c1515b6523444260d8e8f86a71e1c4c8e8a06270c6cbc3fa374ab22 not found: ID does not exist" containerID="f7e0b1923c1515b6523444260d8e8f86a71e1c4c8e8a06270c6cbc3fa374ab22"
Sep 14 10:10:58.011187 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:58.011110    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7e0b1923c1515b6523444260d8e8f86a71e1c4c8e8a06270c6cbc3fa374ab22"} err="failed to get container status \"f7e0b1923c1515b6523444260d8e8f86a71e1c4c8e8a06270c6cbc3fa374ab22\": rpc error: code = NotFound desc = could not find container \"f7e0b1923c1515b6523444260d8e8f86a71e1c4c8e8a06270c6cbc3fa374ab22\": container with ID starting with f7e0b1923c1515b6523444260d8e8f86a71e1c4c8e8a06270c6cbc3fa374ab22 not found: ID does not exist"
Sep 14 10:10:58.013284 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:58.013258    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-2901ec-openshift-default-6dcd9dcd7b-v7r2m"]
Sep 14 10:10:58.633781 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:58.633749    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:10:59.482403 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:10:59.482366    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3188c970-5b86-4c03-95cc-12480aa11882" path="/var/lib/kubelet/pods/3188c970-5b86-4c03-95cc-12480aa11882/volumes"
Sep 14 10:11:01.673277 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:01.673241    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:11:01.748428 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:01.748395    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:11:01.814346 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:01.814310    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:11:01.883943 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:01.883875    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9e19b7-xjf6m"]
Sep 14 10:11:01.886186 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:01.886155    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:11:01.886852 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:01.886829    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9e19b7-xjf6m"]
Sep 14 10:11:01.958374 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:01.958275    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:11:02.238507 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:02.238471    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:11:02.377758 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:02.377722    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:11:03.083614 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:03.083574    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:11:03.483040 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:03.482934    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba8e78fa-5ad0-4ea3-b522-4d49eb6396d5" path="/var/lib/kubelet/pods/ba8e78fa-5ad0-4ea3-b522-4d49eb6396d5/volumes"
Sep 14 10:11:03.914572 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:03.914535    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:11:08.364259 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:08.364224    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:11:08.467025 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:08.466964    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:11:08.627050 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:08.626910    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7"]
Sep 14 10:11:08.627229 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:08.627204    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7" podUID="beb8592b-6036-475a-b404-6acdf015fab2" containerName="istio-proxy" containerID="cri-o://76434b7081486bd266a3f77de96772691feba530594750732057153889a48a8d" gracePeriod=30
Sep 14 10:11:08.651296 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:08.651260    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:11:09.605509 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:09.605464    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:11:09.706366 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:09.706287    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:11:10.200509 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:10.200474    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:11:13.889869 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:13.889782    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7"
Sep 14 10:11:14.001869 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:14.001831    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beb8592b-6036-475a-b404-6acdf015fab2-istio-token\") pod \"beb8592b-6036-475a-b404-6acdf015fab2\" (UID: \"beb8592b-6036-475a-b404-6acdf015fab2\") "
Sep 14 10:11:14.002114 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:14.001890    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beb8592b-6036-475a-b404-6acdf015fab2-workload-socket\") pod \"beb8592b-6036-475a-b404-6acdf015fab2\" (UID: \"beb8592b-6036-475a-b404-6acdf015fab2\") "
Sep 14 10:11:14.002114 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:14.001925    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beb8592b-6036-475a-b404-6acdf015fab2-istiod-ca-cert\") pod \"beb8592b-6036-475a-b404-6acdf015fab2\" (UID: \"beb8592b-6036-475a-b404-6acdf015fab2\") "
Sep 14 10:11:14.002114 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:14.001950    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bq4zj\" (UniqueName: \"kubernetes.io/projected/beb8592b-6036-475a-b404-6acdf015fab2-kube-api-access-bq4zj\") pod \"beb8592b-6036-475a-b404-6acdf015fab2\" (UID: \"beb8592b-6036-475a-b404-6acdf015fab2\") "
Sep 14 10:11:14.002301 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:14.002148    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beb8592b-6036-475a-b404-6acdf015fab2-istio-envoy\") pod \"beb8592b-6036-475a-b404-6acdf015fab2\" (UID: \"beb8592b-6036-475a-b404-6acdf015fab2\") "
Sep 14 10:11:14.002301 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:14.002208    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beb8592b-6036-475a-b404-6acdf015fab2-credential-socket\") pod \"beb8592b-6036-475a-b404-6acdf015fab2\" (UID: \"beb8592b-6036-475a-b404-6acdf015fab2\") "
Sep 14 10:11:14.002301 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:14.002242    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beb8592b-6036-475a-b404-6acdf015fab2-istio-podinfo\") pod \"beb8592b-6036-475a-b404-6acdf015fab2\" (UID: \"beb8592b-6036-475a-b404-6acdf015fab2\") "
Sep 14 10:11:14.002301 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:14.002260    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beb8592b-6036-475a-b404-6acdf015fab2-istio-data\") pod \"beb8592b-6036-475a-b404-6acdf015fab2\" (UID: \"beb8592b-6036-475a-b404-6acdf015fab2\") "
Sep 14 10:11:14.002301 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:14.002256    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beb8592b-6036-475a-b404-6acdf015fab2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "beb8592b-6036-475a-b404-6acdf015fab2" (UID: "beb8592b-6036-475a-b404-6acdf015fab2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:11:14.002301 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:14.002307    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beb8592b-6036-475a-b404-6acdf015fab2-workload-certs\") pod \"beb8592b-6036-475a-b404-6acdf015fab2\" (UID: \"beb8592b-6036-475a-b404-6acdf015fab2\") "
Sep 14 10:11:14.002604 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:14.002436    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/beb8592b-6036-475a-b404-6acdf015fab2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "beb8592b-6036-475a-b404-6acdf015fab2" (UID: "beb8592b-6036-475a-b404-6acdf015fab2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:11:14.002604 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:14.002564    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beb8592b-6036-475a-b404-6acdf015fab2-istio-data" (OuterVolumeSpecName: "istio-data") pod "beb8592b-6036-475a-b404-6acdf015fab2" (UID: "beb8592b-6036-475a-b404-6acdf015fab2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:11:14.002713 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:14.002616    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beb8592b-6036-475a-b404-6acdf015fab2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "beb8592b-6036-475a-b404-6acdf015fab2" (UID: "beb8592b-6036-475a-b404-6acdf015fab2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:11:14.002713 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:14.002638    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beb8592b-6036-475a-b404-6acdf015fab2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "beb8592b-6036-475a-b404-6acdf015fab2" (UID: "beb8592b-6036-475a-b404-6acdf015fab2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:11:14.002828 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:14.002793    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beb8592b-6036-475a-b404-6acdf015fab2-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:11:14.002828 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:14.002813    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beb8592b-6036-475a-b404-6acdf015fab2-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:11:14.002828 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:14.002825    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beb8592b-6036-475a-b404-6acdf015fab2-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:11:14.002980 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:14.002839    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beb8592b-6036-475a-b404-6acdf015fab2-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:11:14.002980 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:14.002851    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beb8592b-6036-475a-b404-6acdf015fab2-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:11:14.004402 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:14.004362    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beb8592b-6036-475a-b404-6acdf015fab2-istio-token" (OuterVolumeSpecName: "istio-token") pod "beb8592b-6036-475a-b404-6acdf015fab2" (UID: "beb8592b-6036-475a-b404-6acdf015fab2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:11:14.004515 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:14.004420    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beb8592b-6036-475a-b404-6acdf015fab2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "beb8592b-6036-475a-b404-6acdf015fab2" (UID: "beb8592b-6036-475a-b404-6acdf015fab2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:11:14.004684 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:14.004659    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/beb8592b-6036-475a-b404-6acdf015fab2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "beb8592b-6036-475a-b404-6acdf015fab2" (UID: "beb8592b-6036-475a-b404-6acdf015fab2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 10:11:14.004791 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:14.004685    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beb8592b-6036-475a-b404-6acdf015fab2-kube-api-access-bq4zj" (OuterVolumeSpecName: "kube-api-access-bq4zj") pod "beb8592b-6036-475a-b404-6acdf015fab2" (UID: "beb8592b-6036-475a-b404-6acdf015fab2"). InnerVolumeSpecName "kube-api-access-bq4zj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:11:14.099075 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:14.099036    2563 generic.go:356] "Generic (PLEG): container finished" podID="beb8592b-6036-475a-b404-6acdf015fab2" containerID="76434b7081486bd266a3f77de96772691feba530594750732057153889a48a8d" exitCode=0
Sep 14 10:11:14.099284 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:14.099102    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7"
Sep 14 10:11:14.099284 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:14.099124    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7" event={"ID":"beb8592b-6036-475a-b404-6acdf015fab2","Type":"ContainerDied","Data":"76434b7081486bd266a3f77de96772691feba530594750732057153889a48a8d"}
Sep 14 10:11:14.099284 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:14.099188    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7" event={"ID":"beb8592b-6036-475a-b404-6acdf015fab2","Type":"ContainerDied","Data":"f0cb02e145830c5dd6dcb6d4571ecfdefe221e5c3fb16aebd7844d2f14a95c44"}
Sep 14 10:11:14.099284 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:14.099214    2563 scope.go:117] "RemoveContainer" containerID="76434b7081486bd266a3f77de96772691feba530594750732057153889a48a8d"
Sep 14 10:11:14.103324 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:14.103292    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beb8592b-6036-475a-b404-6acdf015fab2-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:11:14.103324 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:14.103318    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq4zj\" (UniqueName: \"kubernetes.io/projected/beb8592b-6036-475a-b404-6acdf015fab2-kube-api-access-bq4zj\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:11:14.103324 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:14.103331    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beb8592b-6036-475a-b404-6acdf015fab2-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:11:14.103635 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:14.103340    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beb8592b-6036-475a-b404-6acdf015fab2-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:11:14.110227 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:14.110204    2563 scope.go:117] "RemoveContainer" containerID="f4374eedeb348888a77686228684c05ef1b502504e6f8b380efed384a5d39b64"
Sep 14 10:11:14.120552 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:14.120498    2563 scope.go:117] "RemoveContainer" containerID="76434b7081486bd266a3f77de96772691feba530594750732057153889a48a8d"
Sep 14 10:11:14.120904 ip-10-0-132-4 kubenswrapper[2563]: E0914 10:11:14.120884    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76434b7081486bd266a3f77de96772691feba530594750732057153889a48a8d\": container with ID starting with 76434b7081486bd266a3f77de96772691feba530594750732057153889a48a8d not found: ID does not exist" containerID="76434b7081486bd266a3f77de96772691feba530594750732057153889a48a8d"
Sep 14 10:11:14.120975 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:14.120914    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76434b7081486bd266a3f77de96772691feba530594750732057153889a48a8d"} err="failed to get container status \"76434b7081486bd266a3f77de96772691feba530594750732057153889a48a8d\": rpc error: code = NotFound desc = could not find container \"76434b7081486bd266a3f77de96772691feba530594750732057153889a48a8d\": container with ID starting with 76434b7081486bd266a3f77de96772691feba530594750732057153889a48a8d not found: ID does not exist"
Sep 14 10:11:14.120975 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:14.120934    2563 scope.go:117] "RemoveContainer" containerID="f4374eedeb348888a77686228684c05ef1b502504e6f8b380efed384a5d39b64"
Sep 14 10:11:14.121291 ip-10-0-132-4 kubenswrapper[2563]: E0914 10:11:14.121270    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4374eedeb348888a77686228684c05ef1b502504e6f8b380efed384a5d39b64\": container with ID starting with f4374eedeb348888a77686228684c05ef1b502504e6f8b380efed384a5d39b64 not found: ID does not exist" containerID="f4374eedeb348888a77686228684c05ef1b502504e6f8b380efed384a5d39b64"
Sep 14 10:11:14.121369 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:14.121294    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4374eedeb348888a77686228684c05ef1b502504e6f8b380efed384a5d39b64"} err="failed to get container status \"f4374eedeb348888a77686228684c05ef1b502504e6f8b380efed384a5d39b64\": rpc error: code = NotFound desc = could not find container \"f4374eedeb348888a77686228684c05ef1b502504e6f8b380efed384a5d39b64\": container with ID starting with f4374eedeb348888a77686228684c05ef1b502504e6f8b380efed384a5d39b64 not found: ID does not exist"
Sep 14 10:11:14.122978 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:14.122953    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7"]
Sep 14 10:11:14.129086 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:14.129041    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-9e19b7-openshift-default-794df86f74-97kk7"]
Sep 14 10:11:15.482778 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:15.482725    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="beb8592b-6036-475a-b404-6acdf015fab2" path="/var/lib/kubelet/pods/beb8592b-6036-475a-b404-6acdf015fab2/volumes"
Sep 14 10:11:15.957427 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:15.957391    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:11:31.914521 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:31.914484    2563 scope.go:117] "RemoveContainer" containerID="42d6b78d83b789ba35bf2ffee8b1e2e5b374748a723ca5f503ddec02a546a4ee"
Sep 14 10:11:39.285509 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:39.285473    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7681-434402"]
Sep 14 10:11:39.286058 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:39.286038    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="beb8592b-6036-475a-b404-6acdf015fab2" containerName="istio-proxy"
Sep 14 10:11:39.286167 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:39.286060    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="beb8592b-6036-475a-b404-6acdf015fab2" containerName="istio-proxy"
Sep 14 10:11:39.286167 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:39.286077    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3188c970-5b86-4c03-95cc-12480aa11882" containerName="istio-proxy"
Sep 14 10:11:39.286167 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:39.286085    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3188c970-5b86-4c03-95cc-12480aa11882" containerName="istio-proxy"
Sep 14 10:11:39.286167 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:39.286096    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba8e78fa-5ad0-4ea3-b522-4d49eb6396d5" containerName="revoke-keys"
Sep 14 10:11:39.286167 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:39.286105    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba8e78fa-5ad0-4ea3-b522-4d49eb6396d5" containerName="revoke-keys"
Sep 14 10:11:39.286167 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:39.286115    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3188c970-5b86-4c03-95cc-12480aa11882" containerName="istio-proxy"
Sep 14 10:11:39.286167 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:39.286123    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3188c970-5b86-4c03-95cc-12480aa11882" containerName="istio-proxy"
Sep 14 10:11:39.286513 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:39.286223    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3188c970-5b86-4c03-95cc-12480aa11882" containerName="istio-proxy"
Sep 14 10:11:39.286513 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:39.286236    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="beb8592b-6036-475a-b404-6acdf015fab2" containerName="istio-proxy"
Sep 14 10:11:39.286513 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:39.286246    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3188c970-5b86-4c03-95cc-12480aa11882" containerName="istio-proxy"
Sep 14 10:11:39.286513 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:39.286258    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="beb8592b-6036-475a-b404-6acdf015fab2" containerName="istio-proxy"
Sep 14 10:11:39.286513 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:39.286269    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba8e78fa-5ad0-4ea3-b522-4d49eb6396d5" containerName="revoke-keys"
Sep 14 10:11:39.289437 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:39.289413    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7681-434402"
Sep 14 10:11:39.291713 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:39.291681    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-hjc9w\""
Sep 14 10:11:39.295850 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:39.295818    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7681-434402"]
Sep 14 10:11:39.352226 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:39.352182    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrrwm\" (UniqueName: \"kubernetes.io/projected/28d276c5-d5a3-47f3-8ba2-06356b0ef4eb-kube-api-access-jrrwm\") pod \"test-curl-7681-434402\" (UID: \"28d276c5-d5a3-47f3-8ba2-06356b0ef4eb\") " pod="opendatahub/test-curl-7681-434402"
Sep 14 10:11:39.453763 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:39.453717    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrrwm\" (UniqueName: \"kubernetes.io/projected/28d276c5-d5a3-47f3-8ba2-06356b0ef4eb-kube-api-access-jrrwm\") pod \"test-curl-7681-434402\" (UID: \"28d276c5-d5a3-47f3-8ba2-06356b0ef4eb\") " pod="opendatahub/test-curl-7681-434402"
Sep 14 10:11:39.467328 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:39.467285    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrrwm\" (UniqueName: \"kubernetes.io/projected/28d276c5-d5a3-47f3-8ba2-06356b0ef4eb-kube-api-access-jrrwm\") pod \"test-curl-7681-434402\" (UID: \"28d276c5-d5a3-47f3-8ba2-06356b0ef4eb\") " pod="opendatahub/test-curl-7681-434402"
Sep 14 10:11:39.600883 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:39.600775    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7681-434402"
Sep 14 10:11:39.878884 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:39.878847    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7681-434402"]
Sep 14 10:11:39.880807 ip-10-0-132-4 kubenswrapper[2563]: W0914 10:11:39.880777    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod28d276c5_d5a3_47f3_8ba2_06356b0ef4eb.slice/crio-b38a20834e68c363a3f1f91324965dd3b8e6838a5a036266b06580e6d17f1448 WatchSource:0}: Error finding container b38a20834e68c363a3f1f91324965dd3b8e6838a5a036266b06580e6d17f1448: Status 404 returned error can't find the container with id b38a20834e68c363a3f1f91324965dd3b8e6838a5a036266b06580e6d17f1448
Sep 14 10:11:40.263502 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:40.263458    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7681-434402" event={"ID":"28d276c5-d5a3-47f3-8ba2-06356b0ef4eb","Type":"ContainerStarted","Data":"051b60ef96bb5102a4579e48737cdc0a3f49978a4dfa4342c4bd6e23ce709c93"}
Sep 14 10:11:40.263502 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:40.263505    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7681-434402" event={"ID":"28d276c5-d5a3-47f3-8ba2-06356b0ef4eb","Type":"ContainerStarted","Data":"b38a20834e68c363a3f1f91324965dd3b8e6838a5a036266b06580e6d17f1448"}
Sep 14 10:11:40.280721 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:40.280669    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7681-434402" podStartSLOduration=1.28065319 podStartE2EDuration="1.28065319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:11:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:11:40.279629844 +0000 UTC m=+1689.319304464" watchObservedRunningTime="2026-09-14 10:11:40.28065319 +0000 UTC m=+1689.320327865"
Sep 14 10:11:40.893367 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:40.893320    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7681-434402"]
Sep 14 10:11:40.899357 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:40.899324    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7681-434402"]
Sep 14 10:11:41.269588 ip-10-0-132-4 kubenswrapper[2563]: E0914 10:11:41.269552    2563 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7681-434402" container="test-curl-7681-434402"
Sep 14 10:11:42.247057 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:42.246994    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-51aefd/auth-test-model-51aefd-kserve-7c9dbd4899-87r7z"]
Sep 14 10:11:42.259478 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:42.259447    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:11:42.275448 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:42.275401    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7681-434402" podUID="28d276c5-d5a3-47f3-8ba2-06356b0ef4eb" containerName="test-curl-7681-434402" containerID="cri-o://051b60ef96bb5102a4579e48737cdc0a3f49978a4dfa4342c4bd6e23ce709c93" gracePeriod=2
Sep 14 10:11:43.362135 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:43.362096    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f4d054/auth-test-model-f4d054-kserve-b4655dc8c-4zclc"]
Sep 14 10:11:43.380115 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:43.380081    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:11:44.450863 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:44.450831    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7681-434402"
Sep 14 10:11:44.608446 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:44.608348    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jrrwm\" (UniqueName: \"kubernetes.io/projected/28d276c5-d5a3-47f3-8ba2-06356b0ef4eb-kube-api-access-jrrwm\") pod \"28d276c5-d5a3-47f3-8ba2-06356b0ef4eb\" (UID: \"28d276c5-d5a3-47f3-8ba2-06356b0ef4eb\") "
Sep 14 10:11:44.610547 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:44.610505    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28d276c5-d5a3-47f3-8ba2-06356b0ef4eb-kube-api-access-jrrwm" (OuterVolumeSpecName: "kube-api-access-jrrwm") pod "28d276c5-d5a3-47f3-8ba2-06356b0ef4eb" (UID: "28d276c5-d5a3-47f3-8ba2-06356b0ef4eb"). InnerVolumeSpecName "kube-api-access-jrrwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:11:44.709202 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:44.709159    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrrwm\" (UniqueName: \"kubernetes.io/projected/28d276c5-d5a3-47f3-8ba2-06356b0ef4eb-kube-api-access-jrrwm\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:11:45.294889 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:45.294842    2563 generic.go:356] "Generic (PLEG): container finished" podID="28d276c5-d5a3-47f3-8ba2-06356b0ef4eb" containerID="051b60ef96bb5102a4579e48737cdc0a3f49978a4dfa4342c4bd6e23ce709c93" exitCode=137
Sep 14 10:11:45.294889 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:45.294892    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7681-434402"
Sep 14 10:11:45.295187 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:45.294947    2563 scope.go:117] "RemoveContainer" containerID="051b60ef96bb5102a4579e48737cdc0a3f49978a4dfa4342c4bd6e23ce709c93"
Sep 14 10:11:45.305645 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:45.305623    2563 scope.go:117] "RemoveContainer" containerID="051b60ef96bb5102a4579e48737cdc0a3f49978a4dfa4342c4bd6e23ce709c93"
Sep 14 10:11:45.305984 ip-10-0-132-4 kubenswrapper[2563]: E0914 10:11:45.305962    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"051b60ef96bb5102a4579e48737cdc0a3f49978a4dfa4342c4bd6e23ce709c93\": container with ID starting with 051b60ef96bb5102a4579e48737cdc0a3f49978a4dfa4342c4bd6e23ce709c93 not found: ID does not exist" containerID="051b60ef96bb5102a4579e48737cdc0a3f49978a4dfa4342c4bd6e23ce709c93"
Sep 14 10:11:45.306165 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:45.305994    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"051b60ef96bb5102a4579e48737cdc0a3f49978a4dfa4342c4bd6e23ce709c93"} err="failed to get container status \"051b60ef96bb5102a4579e48737cdc0a3f49978a4dfa4342c4bd6e23ce709c93\": rpc error: code = NotFound desc = could not find container \"051b60ef96bb5102a4579e48737cdc0a3f49978a4dfa4342c4bd6e23ce709c93\": container with ID starting with 051b60ef96bb5102a4579e48737cdc0a3f49978a4dfa4342c4bd6e23ce709c93 not found: ID does not exist"
Sep 14 10:11:45.306394 ip-10-0-132-4 kubenswrapper[2563]: E0914 10:11:45.306374    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7681-434402\" is forbidden: User \"system:node:ip-10-0-132-4.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-4.ec2.internal' and this object" podUID="28d276c5-d5a3-47f3-8ba2-06356b0ef4eb" pod="opendatahub/test-curl-7681-434402"
Sep 14 10:11:45.482240 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:45.482206    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28d276c5-d5a3-47f3-8ba2-06356b0ef4eb" path="/var/lib/kubelet/pods/28d276c5-d5a3-47f3-8ba2-06356b0ef4eb/volumes"
Sep 14 10:11:52.743777 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:52.743734    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:11:54.062224 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:54.062133    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-51aefd/sub-test-model-51aefd-kserve-5b4f744b4c-cjs5l"]
Sep 14 10:11:54.062713 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:54.062692    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="beb8592b-6036-475a-b404-6acdf015fab2" containerName="istio-proxy"
Sep 14 10:11:54.062794 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:54.062716    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="beb8592b-6036-475a-b404-6acdf015fab2" containerName="istio-proxy"
Sep 14 10:11:54.062794 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:54.062740    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28d276c5-d5a3-47f3-8ba2-06356b0ef4eb" containerName="test-curl-7681-434402"
Sep 14 10:11:54.062794 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:54.062751    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="28d276c5-d5a3-47f3-8ba2-06356b0ef4eb" containerName="test-curl-7681-434402"
Sep 14 10:11:54.062949 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:54.062872    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="28d276c5-d5a3-47f3-8ba2-06356b0ef4eb" containerName="test-curl-7681-434402"
Sep 14 10:11:54.067302 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:54.067268    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-51aefd/sub-test-model-51aefd-kserve-5b4f744b4c-cjs5l"
Sep 14 10:11:54.069495 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:54.069462    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-51aefd\"/\"sub-test-model-51aefd-kserve-self-signed-certs\""
Sep 14 10:11:54.073775 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:54.073742    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-51aefd/sub-test-model-51aefd-kserve-5b4f744b4c-cjs5l"]
Sep 14 10:11:54.083880 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:54.083835    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:11:54.212266 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:54.212215    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f0c7af7-e9e0-416f-b907-9edb230a9e48-model-cache\") pod \"sub-test-model-51aefd-kserve-5b4f744b4c-cjs5l\" (UID: \"5f0c7af7-e9e0-416f-b907-9edb230a9e48\") " pod="ai-tenant-e2e-shared-a-w3-51aefd/sub-test-model-51aefd-kserve-5b4f744b4c-cjs5l"
Sep 14 10:11:54.212266 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:54.212275    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f0c7af7-e9e0-416f-b907-9edb230a9e48-tls-certs\") pod \"sub-test-model-51aefd-kserve-5b4f744b4c-cjs5l\" (UID: \"5f0c7af7-e9e0-416f-b907-9edb230a9e48\") " pod="ai-tenant-e2e-shared-a-w3-51aefd/sub-test-model-51aefd-kserve-5b4f744b4c-cjs5l"
Sep 14 10:11:54.212492 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:54.212330    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f0c7af7-e9e0-416f-b907-9edb230a9e48-home\") pod \"sub-test-model-51aefd-kserve-5b4f744b4c-cjs5l\" (UID: \"5f0c7af7-e9e0-416f-b907-9edb230a9e48\") " pod="ai-tenant-e2e-shared-a-w3-51aefd/sub-test-model-51aefd-kserve-5b4f744b4c-cjs5l"
Sep 14 10:11:54.212492 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:54.212375    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f0c7af7-e9e0-416f-b907-9edb230a9e48-tmp-dir\") pod \"sub-test-model-51aefd-kserve-5b4f744b4c-cjs5l\" (UID: \"5f0c7af7-e9e0-416f-b907-9edb230a9e48\") " pod="ai-tenant-e2e-shared-a-w3-51aefd/sub-test-model-51aefd-kserve-5b4f744b4c-cjs5l"
Sep 14 10:11:54.212492 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:54.212395    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f0c7af7-e9e0-416f-b907-9edb230a9e48-dshm\") pod \"sub-test-model-51aefd-kserve-5b4f744b4c-cjs5l\" (UID: \"5f0c7af7-e9e0-416f-b907-9edb230a9e48\") " pod="ai-tenant-e2e-shared-a-w3-51aefd/sub-test-model-51aefd-kserve-5b4f744b4c-cjs5l"
Sep 14 10:11:54.212492 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:54.212475    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx5nj\" (UniqueName: \"kubernetes.io/projected/5f0c7af7-e9e0-416f-b907-9edb230a9e48-kube-api-access-rx5nj\") pod \"sub-test-model-51aefd-kserve-5b4f744b4c-cjs5l\" (UID: \"5f0c7af7-e9e0-416f-b907-9edb230a9e48\") " pod="ai-tenant-e2e-shared-a-w3-51aefd/sub-test-model-51aefd-kserve-5b4f744b4c-cjs5l"
Sep 14 10:11:54.313776 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:54.313663    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx5nj\" (UniqueName: \"kubernetes.io/projected/5f0c7af7-e9e0-416f-b907-9edb230a9e48-kube-api-access-rx5nj\") pod \"sub-test-model-51aefd-kserve-5b4f744b4c-cjs5l\" (UID: \"5f0c7af7-e9e0-416f-b907-9edb230a9e48\") " pod="ai-tenant-e2e-shared-a-w3-51aefd/sub-test-model-51aefd-kserve-5b4f744b4c-cjs5l"
Sep 14 10:11:54.313776 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:54.313755    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f0c7af7-e9e0-416f-b907-9edb230a9e48-model-cache\") pod \"sub-test-model-51aefd-kserve-5b4f744b4c-cjs5l\" (UID: \"5f0c7af7-e9e0-416f-b907-9edb230a9e48\") " pod="ai-tenant-e2e-shared-a-w3-51aefd/sub-test-model-51aefd-kserve-5b4f744b4c-cjs5l"
Sep 14 10:11:54.314046 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:54.313807    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f0c7af7-e9e0-416f-b907-9edb230a9e48-tls-certs\") pod \"sub-test-model-51aefd-kserve-5b4f744b4c-cjs5l\" (UID: \"5f0c7af7-e9e0-416f-b907-9edb230a9e48\") " pod="ai-tenant-e2e-shared-a-w3-51aefd/sub-test-model-51aefd-kserve-5b4f744b4c-cjs5l"
Sep 14 10:11:54.314046 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:54.313856    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f0c7af7-e9e0-416f-b907-9edb230a9e48-home\") pod \"sub-test-model-51aefd-kserve-5b4f744b4c-cjs5l\" (UID: \"5f0c7af7-e9e0-416f-b907-9edb230a9e48\") " pod="ai-tenant-e2e-shared-a-w3-51aefd/sub-test-model-51aefd-kserve-5b4f744b4c-cjs5l"
Sep 14 10:11:54.314046 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:54.313917    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f0c7af7-e9e0-416f-b907-9edb230a9e48-tmp-dir\") pod \"sub-test-model-51aefd-kserve-5b4f744b4c-cjs5l\" (UID: \"5f0c7af7-e9e0-416f-b907-9edb230a9e48\") " pod="ai-tenant-e2e-shared-a-w3-51aefd/sub-test-model-51aefd-kserve-5b4f744b4c-cjs5l"
Sep 14 10:11:54.314046 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:54.313952    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f0c7af7-e9e0-416f-b907-9edb230a9e48-dshm\") pod \"sub-test-model-51aefd-kserve-5b4f744b4c-cjs5l\" (UID: \"5f0c7af7-e9e0-416f-b907-9edb230a9e48\") " pod="ai-tenant-e2e-shared-a-w3-51aefd/sub-test-model-51aefd-kserve-5b4f744b4c-cjs5l"
Sep 14 10:11:54.314406 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:54.314379    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f0c7af7-e9e0-416f-b907-9edb230a9e48-home\") pod \"sub-test-model-51aefd-kserve-5b4f744b4c-cjs5l\" (UID: \"5f0c7af7-e9e0-416f-b907-9edb230a9e48\") " pod="ai-tenant-e2e-shared-a-w3-51aefd/sub-test-model-51aefd-kserve-5b4f744b4c-cjs5l"
Sep 14 10:11:54.314610 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:54.314427    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f0c7af7-e9e0-416f-b907-9edb230a9e48-tmp-dir\") pod \"sub-test-model-51aefd-kserve-5b4f744b4c-cjs5l\" (UID: \"5f0c7af7-e9e0-416f-b907-9edb230a9e48\") " pod="ai-tenant-e2e-shared-a-w3-51aefd/sub-test-model-51aefd-kserve-5b4f744b4c-cjs5l"
Sep 14 10:11:54.314610 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:54.314468    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f0c7af7-e9e0-416f-b907-9edb230a9e48-model-cache\") pod \"sub-test-model-51aefd-kserve-5b4f744b4c-cjs5l\" (UID: \"5f0c7af7-e9e0-416f-b907-9edb230a9e48\") " pod="ai-tenant-e2e-shared-a-w3-51aefd/sub-test-model-51aefd-kserve-5b4f744b4c-cjs5l"
Sep 14 10:11:54.316522 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:54.316484    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f0c7af7-e9e0-416f-b907-9edb230a9e48-dshm\") pod \"sub-test-model-51aefd-kserve-5b4f744b4c-cjs5l\" (UID: \"5f0c7af7-e9e0-416f-b907-9edb230a9e48\") " pod="ai-tenant-e2e-shared-a-w3-51aefd/sub-test-model-51aefd-kserve-5b4f744b4c-cjs5l"
Sep 14 10:11:54.316749 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:54.316731    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f0c7af7-e9e0-416f-b907-9edb230a9e48-tls-certs\") pod \"sub-test-model-51aefd-kserve-5b4f744b4c-cjs5l\" (UID: \"5f0c7af7-e9e0-416f-b907-9edb230a9e48\") " pod="ai-tenant-e2e-shared-a-w3-51aefd/sub-test-model-51aefd-kserve-5b4f744b4c-cjs5l"
Sep 14 10:11:54.323899 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:54.323859    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx5nj\" (UniqueName: \"kubernetes.io/projected/5f0c7af7-e9e0-416f-b907-9edb230a9e48-kube-api-access-rx5nj\") pod \"sub-test-model-51aefd-kserve-5b4f744b4c-cjs5l\" (UID: \"5f0c7af7-e9e0-416f-b907-9edb230a9e48\") " pod="ai-tenant-e2e-shared-a-w3-51aefd/sub-test-model-51aefd-kserve-5b4f744b4c-cjs5l"
Sep 14 10:11:54.380232 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:54.380190    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-51aefd/sub-test-model-51aefd-kserve-5b4f744b4c-cjs5l"
Sep 14 10:11:54.665864 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:54.665833    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-51aefd/sub-test-model-51aefd-kserve-5b4f744b4c-cjs5l"]
Sep 14 10:11:54.667781 ip-10-0-132-4 kubenswrapper[2563]: W0914 10:11:54.667756    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f0c7af7_e9e0_416f_b907_9edb230a9e48.slice/crio-1732f5c9ae8da542df91eb6b31199b09b871ef372bc07f150c0d0833f2ac4e5d WatchSource:0}: Error finding container 1732f5c9ae8da542df91eb6b31199b09b871ef372bc07f150c0d0833f2ac4e5d: Status 404 returned error can't find the container with id 1732f5c9ae8da542df91eb6b31199b09b871ef372bc07f150c0d0833f2ac4e5d
Sep 14 10:11:55.357478 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:55.357427    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-51aefd/sub-test-model-51aefd-kserve-5b4f744b4c-cjs5l" event={"ID":"5f0c7af7-e9e0-416f-b907-9edb230a9e48","Type":"ContainerStarted","Data":"2e596f2483cdd4a18a0dfe9ba4683bcb4bd55cbb5be7e8d37d92e2242896224d"}
Sep 14 10:11:55.357478 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:55.357487    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-51aefd/sub-test-model-51aefd-kserve-5b4f744b4c-cjs5l" event={"ID":"5f0c7af7-e9e0-416f-b907-9edb230a9e48","Type":"ContainerStarted","Data":"1732f5c9ae8da542df91eb6b31199b09b871ef372bc07f150c0d0833f2ac4e5d"}
Sep 14 10:11:55.375330 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:55.375264    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-51aefd/sub-test-model-51aefd-kserve-5b4f744b4c-cjs5l" podStartSLOduration=1.375245502 podStartE2EDuration="1.375245502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:11:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:11:55.373862126 +0000 UTC m=+1704.413536844" watchObservedRunningTime="2026-09-14 10:11:55.375245502 +0000 UTC m=+1704.414920122"
Sep 14 10:11:57.267290 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:57.267249    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-51aefd/auth-test-model-51aefd-kserve-7c9dbd4899-87r7z" podUID="43d9ca79-ce89-424a-bb1b-dd8197cd5d6b" containerName="main" containerID="cri-o://515ce9622fc00fee569d9b68bcbe35504468ba0eb990f127c0de3aca4139c123" gracePeriod=45
Sep 14 10:11:57.533437 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:57.533362    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-51aefd/auth-test-model-51aefd-kserve-7c9dbd4899-87r7z"
Sep 14 10:11:57.542258 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:57.542217    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43d9ca79-ce89-424a-bb1b-dd8197cd5d6b-tls-certs\") pod \"43d9ca79-ce89-424a-bb1b-dd8197cd5d6b\" (UID: \"43d9ca79-ce89-424a-bb1b-dd8197cd5d6b\") "
Sep 14 10:11:57.542467 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:57.542272    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43d9ca79-ce89-424a-bb1b-dd8197cd5d6b-home\") pod \"43d9ca79-ce89-424a-bb1b-dd8197cd5d6b\" (UID: \"43d9ca79-ce89-424a-bb1b-dd8197cd5d6b\") "
Sep 14 10:11:57.542467 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:57.542291    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bktws\" (UniqueName: \"kubernetes.io/projected/43d9ca79-ce89-424a-bb1b-dd8197cd5d6b-kube-api-access-bktws\") pod \"43d9ca79-ce89-424a-bb1b-dd8197cd5d6b\" (UID: \"43d9ca79-ce89-424a-bb1b-dd8197cd5d6b\") "
Sep 14 10:11:57.542467 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:57.542412    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43d9ca79-ce89-424a-bb1b-dd8197cd5d6b-tmp-dir\") pod \"43d9ca79-ce89-424a-bb1b-dd8197cd5d6b\" (UID: \"43d9ca79-ce89-424a-bb1b-dd8197cd5d6b\") "
Sep 14 10:11:57.542644 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:57.542472    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43d9ca79-ce89-424a-bb1b-dd8197cd5d6b-dshm\") pod \"43d9ca79-ce89-424a-bb1b-dd8197cd5d6b\" (UID: \"43d9ca79-ce89-424a-bb1b-dd8197cd5d6b\") "
Sep 14 10:11:57.542644 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:57.542536    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43d9ca79-ce89-424a-bb1b-dd8197cd5d6b-model-cache\") pod \"43d9ca79-ce89-424a-bb1b-dd8197cd5d6b\" (UID: \"43d9ca79-ce89-424a-bb1b-dd8197cd5d6b\") "
Sep 14 10:11:57.542644 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:57.542572    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43d9ca79-ce89-424a-bb1b-dd8197cd5d6b-home" (OuterVolumeSpecName: "home") pod "43d9ca79-ce89-424a-bb1b-dd8197cd5d6b" (UID: "43d9ca79-ce89-424a-bb1b-dd8197cd5d6b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:11:57.542804 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:57.542656    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43d9ca79-ce89-424a-bb1b-dd8197cd5d6b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "43d9ca79-ce89-424a-bb1b-dd8197cd5d6b" (UID: "43d9ca79-ce89-424a-bb1b-dd8197cd5d6b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:11:57.542804 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:57.542791    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43d9ca79-ce89-424a-bb1b-dd8197cd5d6b-model-cache" (OuterVolumeSpecName: "model-cache") pod "43d9ca79-ce89-424a-bb1b-dd8197cd5d6b" (UID: "43d9ca79-ce89-424a-bb1b-dd8197cd5d6b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:11:57.543364 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:57.543071    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43d9ca79-ce89-424a-bb1b-dd8197cd5d6b-tmp-dir\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:11:57.543364 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:57.543095    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43d9ca79-ce89-424a-bb1b-dd8197cd5d6b-model-cache\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:11:57.543364 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:57.543107    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43d9ca79-ce89-424a-bb1b-dd8197cd5d6b-home\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:11:57.544942 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:57.544913    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43d9ca79-ce89-424a-bb1b-dd8197cd5d6b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "43d9ca79-ce89-424a-bb1b-dd8197cd5d6b" (UID: "43d9ca79-ce89-424a-bb1b-dd8197cd5d6b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:11:57.545106 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:57.545056    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43d9ca79-ce89-424a-bb1b-dd8197cd5d6b-kube-api-access-bktws" (OuterVolumeSpecName: "kube-api-access-bktws") pod "43d9ca79-ce89-424a-bb1b-dd8197cd5d6b" (UID: "43d9ca79-ce89-424a-bb1b-dd8197cd5d6b"). InnerVolumeSpecName "kube-api-access-bktws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:11:57.545106 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:57.545077    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43d9ca79-ce89-424a-bb1b-dd8197cd5d6b-dshm" (OuterVolumeSpecName: "dshm") pod "43d9ca79-ce89-424a-bb1b-dd8197cd5d6b" (UID: "43d9ca79-ce89-424a-bb1b-dd8197cd5d6b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:11:57.644447 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:57.644399    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43d9ca79-ce89-424a-bb1b-dd8197cd5d6b-tls-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:11:57.644447 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:57.644434    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bktws\" (UniqueName: \"kubernetes.io/projected/43d9ca79-ce89-424a-bb1b-dd8197cd5d6b-kube-api-access-bktws\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:11:57.644447 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:57.644445    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43d9ca79-ce89-424a-bb1b-dd8197cd5d6b-dshm\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:11:58.377039 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:58.376983    2563 generic.go:356] "Generic (PLEG): container finished" podID="43d9ca79-ce89-424a-bb1b-dd8197cd5d6b" containerID="515ce9622fc00fee569d9b68bcbe35504468ba0eb990f127c0de3aca4139c123" exitCode=0
Sep 14 10:11:58.377456 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:58.377049    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-51aefd/auth-test-model-51aefd-kserve-7c9dbd4899-87r7z"
Sep 14 10:11:58.377456 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:58.377043    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-51aefd/auth-test-model-51aefd-kserve-7c9dbd4899-87r7z" event={"ID":"43d9ca79-ce89-424a-bb1b-dd8197cd5d6b","Type":"ContainerDied","Data":"515ce9622fc00fee569d9b68bcbe35504468ba0eb990f127c0de3aca4139c123"}
Sep 14 10:11:58.377456 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:58.377170    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-51aefd/auth-test-model-51aefd-kserve-7c9dbd4899-87r7z" event={"ID":"43d9ca79-ce89-424a-bb1b-dd8197cd5d6b","Type":"ContainerDied","Data":"74344fc177ab50bc952fa901949e3b440f25cf9fa156faee80407bcace503ff6"}
Sep 14 10:11:58.377456 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:58.377194    2563 scope.go:117] "RemoveContainer" containerID="515ce9622fc00fee569d9b68bcbe35504468ba0eb990f127c0de3aca4139c123"
Sep 14 10:11:58.382492 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:58.382449    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-f4d054/auth-test-model-f4d054-kserve-b4655dc8c-4zclc" podUID="4db918fc-87ba-418e-b581-cad6bed75247" containerName="main" containerID="cri-o://ecb16bd10fe1e9577ada85361634960a380c22d1ee50960227eeb3414ab8025d" gracePeriod=45
Sep 14 10:11:58.389793 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:58.389767    2563 scope.go:117] "RemoveContainer" containerID="515ce9622fc00fee569d9b68bcbe35504468ba0eb990f127c0de3aca4139c123"
Sep 14 10:11:58.390213 ip-10-0-132-4 kubenswrapper[2563]: E0914 10:11:58.390189    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"515ce9622fc00fee569d9b68bcbe35504468ba0eb990f127c0de3aca4139c123\": container with ID starting with 515ce9622fc00fee569d9b68bcbe35504468ba0eb990f127c0de3aca4139c123 not found: ID does not exist" containerID="515ce9622fc00fee569d9b68bcbe35504468ba0eb990f127c0de3aca4139c123"
Sep 14 10:11:58.390310 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:58.390224    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"515ce9622fc00fee569d9b68bcbe35504468ba0eb990f127c0de3aca4139c123"} err="failed to get container status \"515ce9622fc00fee569d9b68bcbe35504468ba0eb990f127c0de3aca4139c123\": rpc error: code = NotFound desc = could not find container \"515ce9622fc00fee569d9b68bcbe35504468ba0eb990f127c0de3aca4139c123\": container with ID starting with 515ce9622fc00fee569d9b68bcbe35504468ba0eb990f127c0de3aca4139c123 not found: ID does not exist"
Sep 14 10:11:58.405070 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:58.405030    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-51aefd/auth-test-model-51aefd-kserve-7c9dbd4899-87r7z"]
Sep 14 10:11:58.409247 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:58.409206    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-51aefd/auth-test-model-51aefd-kserve-7c9dbd4899-87r7z"]
Sep 14 10:11:58.652759 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:58.652713    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f4d054/auth-test-model-f4d054-kserve-b4655dc8c-4zclc"
Sep 14 10:11:58.754941 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:58.754900    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hjvml\" (UniqueName: \"kubernetes.io/projected/4db918fc-87ba-418e-b581-cad6bed75247-kube-api-access-hjvml\") pod \"4db918fc-87ba-418e-b581-cad6bed75247\" (UID: \"4db918fc-87ba-418e-b581-cad6bed75247\") "
Sep 14 10:11:58.755156 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:58.754968    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4db918fc-87ba-418e-b581-cad6bed75247-tmp-dir\") pod \"4db918fc-87ba-418e-b581-cad6bed75247\" (UID: \"4db918fc-87ba-418e-b581-cad6bed75247\") "
Sep 14 10:11:58.755156 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:58.755061    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4db918fc-87ba-418e-b581-cad6bed75247-tls-certs\") pod \"4db918fc-87ba-418e-b581-cad6bed75247\" (UID: \"4db918fc-87ba-418e-b581-cad6bed75247\") "
Sep 14 10:11:58.755156 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:58.755089    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4db918fc-87ba-418e-b581-cad6bed75247-home\") pod \"4db918fc-87ba-418e-b581-cad6bed75247\" (UID: \"4db918fc-87ba-418e-b581-cad6bed75247\") "
Sep 14 10:11:58.755156 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:58.755139    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4db918fc-87ba-418e-b581-cad6bed75247-dshm\") pod \"4db918fc-87ba-418e-b581-cad6bed75247\" (UID: \"4db918fc-87ba-418e-b581-cad6bed75247\") "
Sep 14 10:11:58.755388 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:58.755164    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4db918fc-87ba-418e-b581-cad6bed75247-model-cache\") pod \"4db918fc-87ba-418e-b581-cad6bed75247\" (UID: \"4db918fc-87ba-418e-b581-cad6bed75247\") "
Sep 14 10:11:58.755447 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:58.755417    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4db918fc-87ba-418e-b581-cad6bed75247-home" (OuterVolumeSpecName: "home") pod "4db918fc-87ba-418e-b581-cad6bed75247" (UID: "4db918fc-87ba-418e-b581-cad6bed75247"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:11:58.755495 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:58.755424    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4db918fc-87ba-418e-b581-cad6bed75247-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4db918fc-87ba-418e-b581-cad6bed75247" (UID: "4db918fc-87ba-418e-b581-cad6bed75247"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:11:58.755495 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:58.755441    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4db918fc-87ba-418e-b581-cad6bed75247-model-cache" (OuterVolumeSpecName: "model-cache") pod "4db918fc-87ba-418e-b581-cad6bed75247" (UID: "4db918fc-87ba-418e-b581-cad6bed75247"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:11:58.757546 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:58.757512    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4db918fc-87ba-418e-b581-cad6bed75247-dshm" (OuterVolumeSpecName: "dshm") pod "4db918fc-87ba-418e-b581-cad6bed75247" (UID: "4db918fc-87ba-418e-b581-cad6bed75247"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:11:58.757546 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:58.757526    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4db918fc-87ba-418e-b581-cad6bed75247-kube-api-access-hjvml" (OuterVolumeSpecName: "kube-api-access-hjvml") pod "4db918fc-87ba-418e-b581-cad6bed75247" (UID: "4db918fc-87ba-418e-b581-cad6bed75247"). InnerVolumeSpecName "kube-api-access-hjvml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:11:58.757723 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:58.757511    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4db918fc-87ba-418e-b581-cad6bed75247-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4db918fc-87ba-418e-b581-cad6bed75247" (UID: "4db918fc-87ba-418e-b581-cad6bed75247"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:11:58.856356 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:58.856313    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4db918fc-87ba-418e-b581-cad6bed75247-tmp-dir\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:11:58.856356 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:58.856351    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4db918fc-87ba-418e-b581-cad6bed75247-tls-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:11:58.856356 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:58.856360    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4db918fc-87ba-418e-b581-cad6bed75247-home\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:11:58.856356 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:58.856367    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4db918fc-87ba-418e-b581-cad6bed75247-dshm\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:11:58.856648 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:58.856375    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4db918fc-87ba-418e-b581-cad6bed75247-model-cache\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:11:58.856648 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:58.856385    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjvml\" (UniqueName: \"kubernetes.io/projected/4db918fc-87ba-418e-b581-cad6bed75247-kube-api-access-hjvml\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:11:59.385479 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:59.385437    2563 generic.go:356] "Generic (PLEG): container finished" podID="4db918fc-87ba-418e-b581-cad6bed75247" containerID="ecb16bd10fe1e9577ada85361634960a380c22d1ee50960227eeb3414ab8025d" exitCode=0
Sep 14 10:11:59.385984 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:59.385510    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f4d054/auth-test-model-f4d054-kserve-b4655dc8c-4zclc"
Sep 14 10:11:59.385984 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:59.385517    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f4d054/auth-test-model-f4d054-kserve-b4655dc8c-4zclc" event={"ID":"4db918fc-87ba-418e-b581-cad6bed75247","Type":"ContainerDied","Data":"ecb16bd10fe1e9577ada85361634960a380c22d1ee50960227eeb3414ab8025d"}
Sep 14 10:11:59.385984 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:59.385628    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f4d054/auth-test-model-f4d054-kserve-b4655dc8c-4zclc" event={"ID":"4db918fc-87ba-418e-b581-cad6bed75247","Type":"ContainerDied","Data":"712f17f87ea5035cb57f0b09ace9c01ac880b2b026dc809252c4a86aaab9badd"}
Sep 14 10:11:59.385984 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:59.385646    2563 scope.go:117] "RemoveContainer" containerID="ecb16bd10fe1e9577ada85361634960a380c22d1ee50960227eeb3414ab8025d"
Sep 14 10:11:59.395846 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:59.395824    2563 scope.go:117] "RemoveContainer" containerID="ecb16bd10fe1e9577ada85361634960a380c22d1ee50960227eeb3414ab8025d"
Sep 14 10:11:59.396237 ip-10-0-132-4 kubenswrapper[2563]: E0914 10:11:59.396214    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecb16bd10fe1e9577ada85361634960a380c22d1ee50960227eeb3414ab8025d\": container with ID starting with ecb16bd10fe1e9577ada85361634960a380c22d1ee50960227eeb3414ab8025d not found: ID does not exist" containerID="ecb16bd10fe1e9577ada85361634960a380c22d1ee50960227eeb3414ab8025d"
Sep 14 10:11:59.396314 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:59.396248    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecb16bd10fe1e9577ada85361634960a380c22d1ee50960227eeb3414ab8025d"} err="failed to get container status \"ecb16bd10fe1e9577ada85361634960a380c22d1ee50960227eeb3414ab8025d\": rpc error: code = NotFound desc = could not find container \"ecb16bd10fe1e9577ada85361634960a380c22d1ee50960227eeb3414ab8025d\": container with ID starting with ecb16bd10fe1e9577ada85361634960a380c22d1ee50960227eeb3414ab8025d not found: ID does not exist"
Sep 14 10:11:59.411863 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:59.411824    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f4d054/auth-test-model-f4d054-kserve-b4655dc8c-4zclc"]
Sep 14 10:11:59.414264 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:59.414231    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f4d054/auth-test-model-f4d054-kserve-b4655dc8c-4zclc"]
Sep 14 10:11:59.483595 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:59.483557    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43d9ca79-ce89-424a-bb1b-dd8197cd5d6b" path="/var/lib/kubelet/pods/43d9ca79-ce89-424a-bb1b-dd8197cd5d6b/volumes"
Sep 14 10:11:59.483987 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:11:59.483972    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4db918fc-87ba-418e-b581-cad6bed75247" path="/var/lib/kubelet/pods/4db918fc-87ba-418e-b581-cad6bed75247/volumes"
Sep 14 10:12:04.269211 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:04.269175    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:12:04.380935 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:04.380896    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-51aefd/sub-test-model-51aefd-kserve-5b4f744b4c-cjs5l"
Sep 14 10:12:04.381150 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:04.380955    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-51aefd/sub-test-model-51aefd-kserve-5b4f744b4c-cjs5l"
Sep 14 10:12:04.394138 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:04.394101    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-51aefd/sub-test-model-51aefd-kserve-5b4f744b4c-cjs5l"
Sep 14 10:12:04.428187 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:04.428153    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-51aefd/sub-test-model-51aefd-kserve-5b4f744b4c-cjs5l"
Sep 14 10:12:12.963361 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:12.963309    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-f4d054/sub-test-model-f4d054-kserve-5cc6ff456f-vzc48"]
Sep 14 10:12:12.963994 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:12.963971    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4db918fc-87ba-418e-b581-cad6bed75247" containerName="main"
Sep 14 10:12:12.964089 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:12.964016    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4db918fc-87ba-418e-b581-cad6bed75247" containerName="main"
Sep 14 10:12:12.964089 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:12.964045    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43d9ca79-ce89-424a-bb1b-dd8197cd5d6b" containerName="main"
Sep 14 10:12:12.964089 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:12.964054    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="43d9ca79-ce89-424a-bb1b-dd8197cd5d6b" containerName="main"
Sep 14 10:12:12.964211 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:12.964196    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4db918fc-87ba-418e-b581-cad6bed75247" containerName="main"
Sep 14 10:12:12.964251 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:12.964220    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="43d9ca79-ce89-424a-bb1b-dd8197cd5d6b" containerName="main"
Sep 14 10:12:12.967737 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:12.967712    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f4d054/sub-test-model-f4d054-kserve-5cc6ff456f-vzc48"
Sep 14 10:12:12.970237 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:12.970210    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-f4d054\"/\"openshift-service-ca.crt\""
Sep 14 10:12:12.970237 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:12.970228    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-f4d054\"/\"sub-test-model-f4d054-kserve-self-signed-certs\""
Sep 14 10:12:12.971396 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:12.971367    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-f4d054\"/\"kube-root-ca.crt\""
Sep 14 10:12:12.971619 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:12.971600    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-f4d054\"/\"default-dockercfg-wnn54\""
Sep 14 10:12:12.972806 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:12.972780    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f4d054/sub-test-model-f4d054-kserve-5cc6ff456f-vzc48"]
Sep 14 10:12:12.990855 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:12.990821    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:12:13.092717 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:13.092674    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e128dc2-1f25-4d94-8ff2-943b1d6338bc-dshm\") pod \"sub-test-model-f4d054-kserve-5cc6ff456f-vzc48\" (UID: \"6e128dc2-1f25-4d94-8ff2-943b1d6338bc\") " pod="ai-tenant-e2e-shared-b-w3-f4d054/sub-test-model-f4d054-kserve-5cc6ff456f-vzc48"
Sep 14 10:12:13.092913 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:13.092745    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfcjb\" (UniqueName: \"kubernetes.io/projected/6e128dc2-1f25-4d94-8ff2-943b1d6338bc-kube-api-access-xfcjb\") pod \"sub-test-model-f4d054-kserve-5cc6ff456f-vzc48\" (UID: \"6e128dc2-1f25-4d94-8ff2-943b1d6338bc\") " pod="ai-tenant-e2e-shared-b-w3-f4d054/sub-test-model-f4d054-kserve-5cc6ff456f-vzc48"
Sep 14 10:12:13.092913 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:13.092774    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e128dc2-1f25-4d94-8ff2-943b1d6338bc-tls-certs\") pod \"sub-test-model-f4d054-kserve-5cc6ff456f-vzc48\" (UID: \"6e128dc2-1f25-4d94-8ff2-943b1d6338bc\") " pod="ai-tenant-e2e-shared-b-w3-f4d054/sub-test-model-f4d054-kserve-5cc6ff456f-vzc48"
Sep 14 10:12:13.092913 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:13.092835    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e128dc2-1f25-4d94-8ff2-943b1d6338bc-home\") pod \"sub-test-model-f4d054-kserve-5cc6ff456f-vzc48\" (UID: \"6e128dc2-1f25-4d94-8ff2-943b1d6338bc\") " pod="ai-tenant-e2e-shared-b-w3-f4d054/sub-test-model-f4d054-kserve-5cc6ff456f-vzc48"
Sep 14 10:12:13.092913 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:13.092858    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e128dc2-1f25-4d94-8ff2-943b1d6338bc-model-cache\") pod \"sub-test-model-f4d054-kserve-5cc6ff456f-vzc48\" (UID: \"6e128dc2-1f25-4d94-8ff2-943b1d6338bc\") " pod="ai-tenant-e2e-shared-b-w3-f4d054/sub-test-model-f4d054-kserve-5cc6ff456f-vzc48"
Sep 14 10:12:13.092913 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:13.092912    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e128dc2-1f25-4d94-8ff2-943b1d6338bc-tmp-dir\") pod \"sub-test-model-f4d054-kserve-5cc6ff456f-vzc48\" (UID: \"6e128dc2-1f25-4d94-8ff2-943b1d6338bc\") " pod="ai-tenant-e2e-shared-b-w3-f4d054/sub-test-model-f4d054-kserve-5cc6ff456f-vzc48"
Sep 14 10:12:13.194122 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:13.194073    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e128dc2-1f25-4d94-8ff2-943b1d6338bc-dshm\") pod \"sub-test-model-f4d054-kserve-5cc6ff456f-vzc48\" (UID: \"6e128dc2-1f25-4d94-8ff2-943b1d6338bc\") " pod="ai-tenant-e2e-shared-b-w3-f4d054/sub-test-model-f4d054-kserve-5cc6ff456f-vzc48"
Sep 14 10:12:13.194315 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:13.194174    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfcjb\" (UniqueName: \"kubernetes.io/projected/6e128dc2-1f25-4d94-8ff2-943b1d6338bc-kube-api-access-xfcjb\") pod \"sub-test-model-f4d054-kserve-5cc6ff456f-vzc48\" (UID: \"6e128dc2-1f25-4d94-8ff2-943b1d6338bc\") " pod="ai-tenant-e2e-shared-b-w3-f4d054/sub-test-model-f4d054-kserve-5cc6ff456f-vzc48"
Sep 14 10:12:13.194315 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:13.194210    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e128dc2-1f25-4d94-8ff2-943b1d6338bc-tls-certs\") pod \"sub-test-model-f4d054-kserve-5cc6ff456f-vzc48\" (UID: \"6e128dc2-1f25-4d94-8ff2-943b1d6338bc\") " pod="ai-tenant-e2e-shared-b-w3-f4d054/sub-test-model-f4d054-kserve-5cc6ff456f-vzc48"
Sep 14 10:12:13.194315 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:13.194293    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e128dc2-1f25-4d94-8ff2-943b1d6338bc-home\") pod \"sub-test-model-f4d054-kserve-5cc6ff456f-vzc48\" (UID: \"6e128dc2-1f25-4d94-8ff2-943b1d6338bc\") " pod="ai-tenant-e2e-shared-b-w3-f4d054/sub-test-model-f4d054-kserve-5cc6ff456f-vzc48"
Sep 14 10:12:13.194488 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:13.194328    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e128dc2-1f25-4d94-8ff2-943b1d6338bc-model-cache\") pod \"sub-test-model-f4d054-kserve-5cc6ff456f-vzc48\" (UID: \"6e128dc2-1f25-4d94-8ff2-943b1d6338bc\") " pod="ai-tenant-e2e-shared-b-w3-f4d054/sub-test-model-f4d054-kserve-5cc6ff456f-vzc48"
Sep 14 10:12:13.194488 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:13.194386    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e128dc2-1f25-4d94-8ff2-943b1d6338bc-tmp-dir\") pod \"sub-test-model-f4d054-kserve-5cc6ff456f-vzc48\" (UID: \"6e128dc2-1f25-4d94-8ff2-943b1d6338bc\") " pod="ai-tenant-e2e-shared-b-w3-f4d054/sub-test-model-f4d054-kserve-5cc6ff456f-vzc48"
Sep 14 10:12:13.194716 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:13.194692    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e128dc2-1f25-4d94-8ff2-943b1d6338bc-home\") pod \"sub-test-model-f4d054-kserve-5cc6ff456f-vzc48\" (UID: \"6e128dc2-1f25-4d94-8ff2-943b1d6338bc\") " pod="ai-tenant-e2e-shared-b-w3-f4d054/sub-test-model-f4d054-kserve-5cc6ff456f-vzc48"
Sep 14 10:12:13.194782 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:13.194733    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e128dc2-1f25-4d94-8ff2-943b1d6338bc-tmp-dir\") pod \"sub-test-model-f4d054-kserve-5cc6ff456f-vzc48\" (UID: \"6e128dc2-1f25-4d94-8ff2-943b1d6338bc\") " pod="ai-tenant-e2e-shared-b-w3-f4d054/sub-test-model-f4d054-kserve-5cc6ff456f-vzc48"
Sep 14 10:12:13.194827 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:13.194804    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e128dc2-1f25-4d94-8ff2-943b1d6338bc-model-cache\") pod \"sub-test-model-f4d054-kserve-5cc6ff456f-vzc48\" (UID: \"6e128dc2-1f25-4d94-8ff2-943b1d6338bc\") " pod="ai-tenant-e2e-shared-b-w3-f4d054/sub-test-model-f4d054-kserve-5cc6ff456f-vzc48"
Sep 14 10:12:13.196597 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:13.196574    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e128dc2-1f25-4d94-8ff2-943b1d6338bc-dshm\") pod \"sub-test-model-f4d054-kserve-5cc6ff456f-vzc48\" (UID: \"6e128dc2-1f25-4d94-8ff2-943b1d6338bc\") " pod="ai-tenant-e2e-shared-b-w3-f4d054/sub-test-model-f4d054-kserve-5cc6ff456f-vzc48"
Sep 14 10:12:13.196811 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:13.196795    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e128dc2-1f25-4d94-8ff2-943b1d6338bc-tls-certs\") pod \"sub-test-model-f4d054-kserve-5cc6ff456f-vzc48\" (UID: \"6e128dc2-1f25-4d94-8ff2-943b1d6338bc\") " pod="ai-tenant-e2e-shared-b-w3-f4d054/sub-test-model-f4d054-kserve-5cc6ff456f-vzc48"
Sep 14 10:12:13.202611 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:13.202572    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfcjb\" (UniqueName: \"kubernetes.io/projected/6e128dc2-1f25-4d94-8ff2-943b1d6338bc-kube-api-access-xfcjb\") pod \"sub-test-model-f4d054-kserve-5cc6ff456f-vzc48\" (UID: \"6e128dc2-1f25-4d94-8ff2-943b1d6338bc\") " pod="ai-tenant-e2e-shared-b-w3-f4d054/sub-test-model-f4d054-kserve-5cc6ff456f-vzc48"
Sep 14 10:12:13.280340 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:13.280298    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f4d054/sub-test-model-f4d054-kserve-5cc6ff456f-vzc48"
Sep 14 10:12:13.559427 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:13.559398    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f4d054/sub-test-model-f4d054-kserve-5cc6ff456f-vzc48"]
Sep 14 10:12:13.561198 ip-10-0-132-4 kubenswrapper[2563]: W0914 10:12:13.561164    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6e128dc2_1f25_4d94_8ff2_943b1d6338bc.slice/crio-5f4d5675b175b4d21658495cde7bdec14323e50b8ecdcb7ee959a893e7d60818 WatchSource:0}: Error finding container 5f4d5675b175b4d21658495cde7bdec14323e50b8ecdcb7ee959a893e7d60818: Status 404 returned error can't find the container with id 5f4d5675b175b4d21658495cde7bdec14323e50b8ecdcb7ee959a893e7d60818
Sep 14 10:12:14.486955 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:14.486915    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f4d054/sub-test-model-f4d054-kserve-5cc6ff456f-vzc48" event={"ID":"6e128dc2-1f25-4d94-8ff2-943b1d6338bc","Type":"ContainerStarted","Data":"a82895699de9f19d8d7009d494df095684654ec7dd0f217fd52787653dd46572"}
Sep 14 10:12:14.487417 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:14.486966    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f4d054/sub-test-model-f4d054-kserve-5cc6ff456f-vzc48" event={"ID":"6e128dc2-1f25-4d94-8ff2-943b1d6338bc","Type":"ContainerStarted","Data":"5f4d5675b175b4d21658495cde7bdec14323e50b8ecdcb7ee959a893e7d60818"}
Sep 14 10:12:14.505714 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:14.505654    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-f4d054/sub-test-model-f4d054-kserve-5cc6ff456f-vzc48" podStartSLOduration=2.505629282 podStartE2EDuration="2.505629282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:12:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:12:14.504263995 +0000 UTC m=+1723.543938627" watchObservedRunningTime="2026-09-14 10:12:14.505629282 +0000 UTC m=+1723.545303935"
Sep 14 10:12:23.281227 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:23.281177    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-f4d054/sub-test-model-f4d054-kserve-5cc6ff456f-vzc48"
Sep 14 10:12:23.281653 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:23.281248    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-f4d054/sub-test-model-f4d054-kserve-5cc6ff456f-vzc48"
Sep 14 10:12:23.294120 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:23.294083    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-f4d054/sub-test-model-f4d054-kserve-5cc6ff456f-vzc48"
Sep 14 10:12:23.552419 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:23.552324    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-f4d054/sub-test-model-f4d054-kserve-5cc6ff456f-vzc48"
Sep 14 10:12:47.403674 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:47.403633    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7681-045dba"]
Sep 14 10:12:47.407483 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:47.407457    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7681-045dba"
Sep 14 10:12:47.411190 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:47.411148    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-hjc9w\""
Sep 14 10:12:47.413293 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:47.413243    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7681-045dba"]
Sep 14 10:12:47.537455 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:47.537414    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lccch\" (UniqueName: \"kubernetes.io/projected/354b5fc3-a236-47f4-aa97-eb3e1be25e4c-kube-api-access-lccch\") pod \"test-curl-7681-045dba\" (UID: \"354b5fc3-a236-47f4-aa97-eb3e1be25e4c\") " pod="opendatahub/test-curl-7681-045dba"
Sep 14 10:12:47.638488 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:47.638423    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lccch\" (UniqueName: \"kubernetes.io/projected/354b5fc3-a236-47f4-aa97-eb3e1be25e4c-kube-api-access-lccch\") pod \"test-curl-7681-045dba\" (UID: \"354b5fc3-a236-47f4-aa97-eb3e1be25e4c\") " pod="opendatahub/test-curl-7681-045dba"
Sep 14 10:12:47.646991 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:47.646961    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lccch\" (UniqueName: \"kubernetes.io/projected/354b5fc3-a236-47f4-aa97-eb3e1be25e4c-kube-api-access-lccch\") pod \"test-curl-7681-045dba\" (UID: \"354b5fc3-a236-47f4-aa97-eb3e1be25e4c\") " pod="opendatahub/test-curl-7681-045dba"
Sep 14 10:12:47.719131 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:47.719029    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7681-045dba"
Sep 14 10:12:47.995332 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:47.995245    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7681-045dba"]
Sep 14 10:12:47.998196 ip-10-0-132-4 kubenswrapper[2563]: W0914 10:12:47.998154    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod354b5fc3_a236_47f4_aa97_eb3e1be25e4c.slice/crio-57835fcdef4c88801af0a920fe9821ad8abf7f5cc671b4bb488f7ae9b2dca021 WatchSource:0}: Error finding container 57835fcdef4c88801af0a920fe9821ad8abf7f5cc671b4bb488f7ae9b2dca021: Status 404 returned error can't find the container with id 57835fcdef4c88801af0a920fe9821ad8abf7f5cc671b4bb488f7ae9b2dca021
Sep 14 10:12:48.693100 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:48.693055    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7681-045dba" event={"ID":"354b5fc3-a236-47f4-aa97-eb3e1be25e4c","Type":"ContainerStarted","Data":"ef9df7e494857d259204d52192258f3056f315a9d99833c52ca30adc8e66fc3e"}
Sep 14 10:12:48.693503 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:48.693110    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7681-045dba" event={"ID":"354b5fc3-a236-47f4-aa97-eb3e1be25e4c","Type":"ContainerStarted","Data":"57835fcdef4c88801af0a920fe9821ad8abf7f5cc671b4bb488f7ae9b2dca021"}
Sep 14 10:12:48.713348 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:48.713280    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7681-045dba" podStartSLOduration=1.713256256 podStartE2EDuration="1.713256256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:12:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:12:48.70977733 +0000 UTC m=+1757.749452005" watchObservedRunningTime="2026-09-14 10:12:48.713256256 +0000 UTC m=+1757.752930898"
Sep 14 10:12:49.305459 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:49.305418    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7681-045dba"]
Sep 14 10:12:49.309670 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:49.309628    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7681-045dba"]
Sep 14 10:12:49.419801 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:49.419661    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7681-49d9b0"]
Sep 14 10:12:49.420134 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:49.420115    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="354b5fc3-a236-47f4-aa97-eb3e1be25e4c" containerName="test-curl-7681-045dba"
Sep 14 10:12:49.420200 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:49.420137    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="354b5fc3-a236-47f4-aa97-eb3e1be25e4c" containerName="test-curl-7681-045dba"
Sep 14 10:12:49.420243 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:49.420210    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="354b5fc3-a236-47f4-aa97-eb3e1be25e4c" containerName="test-curl-7681-045dba"
Sep 14 10:12:49.423492 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:49.423467    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7681-49d9b0"
Sep 14 10:12:49.431299 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:49.431268    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7681-49d9b0"]
Sep 14 10:12:49.557245 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:49.557204    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmr6x\" (UniqueName: \"kubernetes.io/projected/69b6054a-c829-492e-a354-dfec6fefc490-kube-api-access-fmr6x\") pod \"test-curl-7681-49d9b0\" (UID: \"69b6054a-c829-492e-a354-dfec6fefc490\") " pod="opendatahub/test-curl-7681-49d9b0"
Sep 14 10:12:49.658096 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:49.658054    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmr6x\" (UniqueName: \"kubernetes.io/projected/69b6054a-c829-492e-a354-dfec6fefc490-kube-api-access-fmr6x\") pod \"test-curl-7681-49d9b0\" (UID: \"69b6054a-c829-492e-a354-dfec6fefc490\") " pod="opendatahub/test-curl-7681-49d9b0"
Sep 14 10:12:49.668092 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:49.668047    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmr6x\" (UniqueName: \"kubernetes.io/projected/69b6054a-c829-492e-a354-dfec6fefc490-kube-api-access-fmr6x\") pod \"test-curl-7681-49d9b0\" (UID: \"69b6054a-c829-492e-a354-dfec6fefc490\") " pod="opendatahub/test-curl-7681-49d9b0"
Sep 14 10:12:49.699575 ip-10-0-132-4 kubenswrapper[2563]: E0914 10:12:49.699480    2563 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7681-045dba" container="test-curl-7681-045dba"
Sep 14 10:12:49.737467 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:49.737432    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7681-49d9b0"
Sep 14 10:12:50.011962 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:50.011929    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7681-49d9b0"]
Sep 14 10:12:50.013733 ip-10-0-132-4 kubenswrapper[2563]: W0914 10:12:50.013700    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod69b6054a_c829_492e_a354_dfec6fefc490.slice/crio-fc2ecdadbb47c600ee3a2e6dae20b64486074956efc31a469f1cd8c4fab3a799 WatchSource:0}: Error finding container fc2ecdadbb47c600ee3a2e6dae20b64486074956efc31a469f1cd8c4fab3a799: Status 404 returned error can't find the container with id fc2ecdadbb47c600ee3a2e6dae20b64486074956efc31a469f1cd8c4fab3a799
Sep 14 10:12:50.707366 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:50.707326    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7681-49d9b0" event={"ID":"69b6054a-c829-492e-a354-dfec6fefc490","Type":"ContainerStarted","Data":"076c7dc8db883ce0dc0bbbbad9556a8abd0e0e8776a2fc3c068304d93684949d"}
Sep 14 10:12:50.707845 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:50.707376    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7681-49d9b0" event={"ID":"69b6054a-c829-492e-a354-dfec6fefc490","Type":"ContainerStarted","Data":"fc2ecdadbb47c600ee3a2e6dae20b64486074956efc31a469f1cd8c4fab3a799"}
Sep 14 10:12:50.707845 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:50.707477    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7681-045dba" podUID="354b5fc3-a236-47f4-aa97-eb3e1be25e4c" containerName="test-curl-7681-045dba" containerID="cri-o://ef9df7e494857d259204d52192258f3056f315a9d99833c52ca30adc8e66fc3e" gracePeriod=2
Sep 14 10:12:50.723727 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:50.723660    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7681-49d9b0" podStartSLOduration=1.723639801 podStartE2EDuration="1.723639801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:12:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:12:50.721110626 +0000 UTC m=+1759.760785249" watchObservedRunningTime="2026-09-14 10:12:50.723639801 +0000 UTC m=+1759.763314423"
Sep 14 10:12:51.311135 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:51.311092    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7681-49d9b0"]
Sep 14 10:12:51.315787 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:51.315748    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7681-49d9b0"]
Sep 14 10:12:51.714090 ip-10-0-132-4 kubenswrapper[2563]: E0914 10:12:51.714058    2563 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7681-49d9b0" container="test-curl-7681-49d9b0"
Sep 14 10:12:52.720991 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:52.720940    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7681-49d9b0" podUID="69b6054a-c829-492e-a354-dfec6fefc490" containerName="test-curl-7681-49d9b0" containerID="cri-o://076c7dc8db883ce0dc0bbbbad9556a8abd0e0e8776a2fc3c068304d93684949d" gracePeriod=2
Sep 14 10:12:52.903267 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:52.903227    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7681-045dba"
Sep 14 10:12:52.995247 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:52.995210    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lccch\" (UniqueName: \"kubernetes.io/projected/354b5fc3-a236-47f4-aa97-eb3e1be25e4c-kube-api-access-lccch\") pod \"354b5fc3-a236-47f4-aa97-eb3e1be25e4c\" (UID: \"354b5fc3-a236-47f4-aa97-eb3e1be25e4c\") "
Sep 14 10:12:52.997463 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:52.997430    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/354b5fc3-a236-47f4-aa97-eb3e1be25e4c-kube-api-access-lccch" (OuterVolumeSpecName: "kube-api-access-lccch") pod "354b5fc3-a236-47f4-aa97-eb3e1be25e4c" (UID: "354b5fc3-a236-47f4-aa97-eb3e1be25e4c"). InnerVolumeSpecName "kube-api-access-lccch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:12:53.096465 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:53.096415    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lccch\" (UniqueName: \"kubernetes.io/projected/354b5fc3-a236-47f4-aa97-eb3e1be25e4c-kube-api-access-lccch\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:12:53.484419 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:53.484383    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="354b5fc3-a236-47f4-aa97-eb3e1be25e4c" path="/var/lib/kubelet/pods/354b5fc3-a236-47f4-aa97-eb3e1be25e4c/volumes"
Sep 14 10:12:53.484887 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:53.484866    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-51aefd/sub-test-model-51aefd-kserve-5b4f744b4c-cjs5l"]
Sep 14 10:12:53.494297 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:53.494262    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:12:53.728954 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:53.728917    2563 generic.go:356] "Generic (PLEG): container finished" podID="354b5fc3-a236-47f4-aa97-eb3e1be25e4c" containerID="ef9df7e494857d259204d52192258f3056f315a9d99833c52ca30adc8e66fc3e" exitCode=137
Sep 14 10:12:53.729409 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:53.728964    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7681-045dba"
Sep 14 10:12:53.729409 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:53.728995    2563 scope.go:117] "RemoveContainer" containerID="ef9df7e494857d259204d52192258f3056f315a9d99833c52ca30adc8e66fc3e"
Sep 14 10:12:53.738871 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:53.738848    2563 scope.go:117] "RemoveContainer" containerID="ef9df7e494857d259204d52192258f3056f315a9d99833c52ca30adc8e66fc3e"
Sep 14 10:12:53.739299 ip-10-0-132-4 kubenswrapper[2563]: E0914 10:12:53.739275    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef9df7e494857d259204d52192258f3056f315a9d99833c52ca30adc8e66fc3e\": container with ID starting with ef9df7e494857d259204d52192258f3056f315a9d99833c52ca30adc8e66fc3e not found: ID does not exist" containerID="ef9df7e494857d259204d52192258f3056f315a9d99833c52ca30adc8e66fc3e"
Sep 14 10:12:53.739404 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:53.739309    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef9df7e494857d259204d52192258f3056f315a9d99833c52ca30adc8e66fc3e"} err="failed to get container status \"ef9df7e494857d259204d52192258f3056f315a9d99833c52ca30adc8e66fc3e\": rpc error: code = NotFound desc = could not find container \"ef9df7e494857d259204d52192258f3056f315a9d99833c52ca30adc8e66fc3e\": container with ID starting with ef9df7e494857d259204d52192258f3056f315a9d99833c52ca30adc8e66fc3e not found: ID does not exist"
Sep 14 10:12:54.370338 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:54.370295    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f4d054/sub-test-model-f4d054-kserve-5cc6ff456f-vzc48"]
Sep 14 10:12:54.393200 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:54.393161    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:12:54.897085 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:54.897052    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7681-49d9b0"
Sep 14 10:12:55.013337 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:55.013299    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fmr6x\" (UniqueName: \"kubernetes.io/projected/69b6054a-c829-492e-a354-dfec6fefc490-kube-api-access-fmr6x\") pod \"69b6054a-c829-492e-a354-dfec6fefc490\" (UID: \"69b6054a-c829-492e-a354-dfec6fefc490\") "
Sep 14 10:12:55.015460 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:55.015429    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69b6054a-c829-492e-a354-dfec6fefc490-kube-api-access-fmr6x" (OuterVolumeSpecName: "kube-api-access-fmr6x") pod "69b6054a-c829-492e-a354-dfec6fefc490" (UID: "69b6054a-c829-492e-a354-dfec6fefc490"). InnerVolumeSpecName "kube-api-access-fmr6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:12:55.114523 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:55.114481    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmr6x\" (UniqueName: \"kubernetes.io/projected/69b6054a-c829-492e-a354-dfec6fefc490-kube-api-access-fmr6x\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:12:55.482884 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:55.482791    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69b6054a-c829-492e-a354-dfec6fefc490" path="/var/lib/kubelet/pods/69b6054a-c829-492e-a354-dfec6fefc490/volumes"
Sep 14 10:12:55.744508 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:55.744468    2563 generic.go:356] "Generic (PLEG): container finished" podID="69b6054a-c829-492e-a354-dfec6fefc490" containerID="076c7dc8db883ce0dc0bbbbad9556a8abd0e0e8776a2fc3c068304d93684949d" exitCode=137
Sep 14 10:12:55.744718 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:55.744517    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7681-49d9b0"
Sep 14 10:12:55.744718 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:55.744563    2563 scope.go:117] "RemoveContainer" containerID="076c7dc8db883ce0dc0bbbbad9556a8abd0e0e8776a2fc3c068304d93684949d"
Sep 14 10:12:55.755680 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:55.755532    2563 scope.go:117] "RemoveContainer" containerID="076c7dc8db883ce0dc0bbbbad9556a8abd0e0e8776a2fc3c068304d93684949d"
Sep 14 10:12:55.756320 ip-10-0-132-4 kubenswrapper[2563]: E0914 10:12:55.756247    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"076c7dc8db883ce0dc0bbbbad9556a8abd0e0e8776a2fc3c068304d93684949d\": container with ID starting with 076c7dc8db883ce0dc0bbbbad9556a8abd0e0e8776a2fc3c068304d93684949d not found: ID does not exist" containerID="076c7dc8db883ce0dc0bbbbad9556a8abd0e0e8776a2fc3c068304d93684949d"
Sep 14 10:12:55.756320 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:55.756285    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"076c7dc8db883ce0dc0bbbbad9556a8abd0e0e8776a2fc3c068304d93684949d"} err="failed to get container status \"076c7dc8db883ce0dc0bbbbad9556a8abd0e0e8776a2fc3c068304d93684949d\": rpc error: code = NotFound desc = could not find container \"076c7dc8db883ce0dc0bbbbad9556a8abd0e0e8776a2fc3c068304d93684949d\": container with ID starting with 076c7dc8db883ce0dc0bbbbad9556a8abd0e0e8776a2fc3c068304d93684949d not found: ID does not exist"
Sep 14 10:12:59.760818 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:59.760780    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-51aefd/rate-test-model-51aefd-kserve-5d5f889dc5-949bj"]
Sep 14 10:12:59.761261 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:59.761200    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69b6054a-c829-492e-a354-dfec6fefc490" containerName="test-curl-7681-49d9b0"
Sep 14 10:12:59.761261 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:59.761219    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="69b6054a-c829-492e-a354-dfec6fefc490" containerName="test-curl-7681-49d9b0"
Sep 14 10:12:59.761348 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:59.761338    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="69b6054a-c829-492e-a354-dfec6fefc490" containerName="test-curl-7681-49d9b0"
Sep 14 10:12:59.766030 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:59.765977    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-51aefd/rate-test-model-51aefd-kserve-5d5f889dc5-949bj"
Sep 14 10:12:59.768269 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:59.768234    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-51aefd\"/\"rate-test-model-51aefd-kserve-self-signed-certs\""
Sep 14 10:12:59.775100 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:59.775024    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-51aefd/rate-test-model-51aefd-kserve-5d5f889dc5-949bj"]
Sep 14 10:12:59.802167 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:59.802132    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:12:59.859946 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:59.859903    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/209ae11a-204c-48ff-83ef-10f4c073290f-home\") pod \"rate-test-model-51aefd-kserve-5d5f889dc5-949bj\" (UID: \"209ae11a-204c-48ff-83ef-10f4c073290f\") " pod="ai-tenant-e2e-shared-a-w3-51aefd/rate-test-model-51aefd-kserve-5d5f889dc5-949bj"
Sep 14 10:12:59.860134 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:59.859967    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/209ae11a-204c-48ff-83ef-10f4c073290f-tmp-dir\") pod \"rate-test-model-51aefd-kserve-5d5f889dc5-949bj\" (UID: \"209ae11a-204c-48ff-83ef-10f4c073290f\") " pod="ai-tenant-e2e-shared-a-w3-51aefd/rate-test-model-51aefd-kserve-5d5f889dc5-949bj"
Sep 14 10:12:59.860134 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:59.860062    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/209ae11a-204c-48ff-83ef-10f4c073290f-dshm\") pod \"rate-test-model-51aefd-kserve-5d5f889dc5-949bj\" (UID: \"209ae11a-204c-48ff-83ef-10f4c073290f\") " pod="ai-tenant-e2e-shared-a-w3-51aefd/rate-test-model-51aefd-kserve-5d5f889dc5-949bj"
Sep 14 10:12:59.860134 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:59.860119    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/209ae11a-204c-48ff-83ef-10f4c073290f-tls-certs\") pod \"rate-test-model-51aefd-kserve-5d5f889dc5-949bj\" (UID: \"209ae11a-204c-48ff-83ef-10f4c073290f\") " pod="ai-tenant-e2e-shared-a-w3-51aefd/rate-test-model-51aefd-kserve-5d5f889dc5-949bj"
Sep 14 10:12:59.860314 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:59.860205    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/209ae11a-204c-48ff-83ef-10f4c073290f-model-cache\") pod \"rate-test-model-51aefd-kserve-5d5f889dc5-949bj\" (UID: \"209ae11a-204c-48ff-83ef-10f4c073290f\") " pod="ai-tenant-e2e-shared-a-w3-51aefd/rate-test-model-51aefd-kserve-5d5f889dc5-949bj"
Sep 14 10:12:59.860314 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:59.860247    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw6n5\" (UniqueName: \"kubernetes.io/projected/209ae11a-204c-48ff-83ef-10f4c073290f-kube-api-access-mw6n5\") pod \"rate-test-model-51aefd-kserve-5d5f889dc5-949bj\" (UID: \"209ae11a-204c-48ff-83ef-10f4c073290f\") " pod="ai-tenant-e2e-shared-a-w3-51aefd/rate-test-model-51aefd-kserve-5d5f889dc5-949bj"
Sep 14 10:12:59.961323 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:59.961274    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/209ae11a-204c-48ff-83ef-10f4c073290f-model-cache\") pod \"rate-test-model-51aefd-kserve-5d5f889dc5-949bj\" (UID: \"209ae11a-204c-48ff-83ef-10f4c073290f\") " pod="ai-tenant-e2e-shared-a-w3-51aefd/rate-test-model-51aefd-kserve-5d5f889dc5-949bj"
Sep 14 10:12:59.961538 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:59.961333    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw6n5\" (UniqueName: \"kubernetes.io/projected/209ae11a-204c-48ff-83ef-10f4c073290f-kube-api-access-mw6n5\") pod \"rate-test-model-51aefd-kserve-5d5f889dc5-949bj\" (UID: \"209ae11a-204c-48ff-83ef-10f4c073290f\") " pod="ai-tenant-e2e-shared-a-w3-51aefd/rate-test-model-51aefd-kserve-5d5f889dc5-949bj"
Sep 14 10:12:59.961538 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:59.961362    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/209ae11a-204c-48ff-83ef-10f4c073290f-home\") pod \"rate-test-model-51aefd-kserve-5d5f889dc5-949bj\" (UID: \"209ae11a-204c-48ff-83ef-10f4c073290f\") " pod="ai-tenant-e2e-shared-a-w3-51aefd/rate-test-model-51aefd-kserve-5d5f889dc5-949bj"
Sep 14 10:12:59.961538 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:59.961418    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/209ae11a-204c-48ff-83ef-10f4c073290f-tmp-dir\") pod \"rate-test-model-51aefd-kserve-5d5f889dc5-949bj\" (UID: \"209ae11a-204c-48ff-83ef-10f4c073290f\") " pod="ai-tenant-e2e-shared-a-w3-51aefd/rate-test-model-51aefd-kserve-5d5f889dc5-949bj"
Sep 14 10:12:59.961538 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:59.961469    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/209ae11a-204c-48ff-83ef-10f4c073290f-dshm\") pod \"rate-test-model-51aefd-kserve-5d5f889dc5-949bj\" (UID: \"209ae11a-204c-48ff-83ef-10f4c073290f\") " pod="ai-tenant-e2e-shared-a-w3-51aefd/rate-test-model-51aefd-kserve-5d5f889dc5-949bj"
Sep 14 10:12:59.961538 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:59.961534    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/209ae11a-204c-48ff-83ef-10f4c073290f-tls-certs\") pod \"rate-test-model-51aefd-kserve-5d5f889dc5-949bj\" (UID: \"209ae11a-204c-48ff-83ef-10f4c073290f\") " pod="ai-tenant-e2e-shared-a-w3-51aefd/rate-test-model-51aefd-kserve-5d5f889dc5-949bj"
Sep 14 10:12:59.961788 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:59.961764    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/209ae11a-204c-48ff-83ef-10f4c073290f-model-cache\") pod \"rate-test-model-51aefd-kserve-5d5f889dc5-949bj\" (UID: \"209ae11a-204c-48ff-83ef-10f4c073290f\") " pod="ai-tenant-e2e-shared-a-w3-51aefd/rate-test-model-51aefd-kserve-5d5f889dc5-949bj"
Sep 14 10:12:59.961850 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:59.961783    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/209ae11a-204c-48ff-83ef-10f4c073290f-home\") pod \"rate-test-model-51aefd-kserve-5d5f889dc5-949bj\" (UID: \"209ae11a-204c-48ff-83ef-10f4c073290f\") " pod="ai-tenant-e2e-shared-a-w3-51aefd/rate-test-model-51aefd-kserve-5d5f889dc5-949bj"
Sep 14 10:12:59.961850 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:59.961817    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/209ae11a-204c-48ff-83ef-10f4c073290f-tmp-dir\") pod \"rate-test-model-51aefd-kserve-5d5f889dc5-949bj\" (UID: \"209ae11a-204c-48ff-83ef-10f4c073290f\") " pod="ai-tenant-e2e-shared-a-w3-51aefd/rate-test-model-51aefd-kserve-5d5f889dc5-949bj"
Sep 14 10:12:59.963830 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:59.963793    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/209ae11a-204c-48ff-83ef-10f4c073290f-dshm\") pod \"rate-test-model-51aefd-kserve-5d5f889dc5-949bj\" (UID: \"209ae11a-204c-48ff-83ef-10f4c073290f\") " pod="ai-tenant-e2e-shared-a-w3-51aefd/rate-test-model-51aefd-kserve-5d5f889dc5-949bj"
Sep 14 10:12:59.964134 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:59.964115    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/209ae11a-204c-48ff-83ef-10f4c073290f-tls-certs\") pod \"rate-test-model-51aefd-kserve-5d5f889dc5-949bj\" (UID: \"209ae11a-204c-48ff-83ef-10f4c073290f\") " pod="ai-tenant-e2e-shared-a-w3-51aefd/rate-test-model-51aefd-kserve-5d5f889dc5-949bj"
Sep 14 10:12:59.970224 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:12:59.970187    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw6n5\" (UniqueName: \"kubernetes.io/projected/209ae11a-204c-48ff-83ef-10f4c073290f-kube-api-access-mw6n5\") pod \"rate-test-model-51aefd-kserve-5d5f889dc5-949bj\" (UID: \"209ae11a-204c-48ff-83ef-10f4c073290f\") " pod="ai-tenant-e2e-shared-a-w3-51aefd/rate-test-model-51aefd-kserve-5d5f889dc5-949bj"
Sep 14 10:13:00.078158 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:00.078047    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-51aefd/rate-test-model-51aefd-kserve-5d5f889dc5-949bj"
Sep 14 10:13:00.359076 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:00.359042    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-51aefd/rate-test-model-51aefd-kserve-5d5f889dc5-949bj"]
Sep 14 10:13:00.360796 ip-10-0-132-4 kubenswrapper[2563]: W0914 10:13:00.360764    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod209ae11a_204c_48ff_83ef_10f4c073290f.slice/crio-f1546d060823a1dc89c2d22a70d0d04a6a5bbb4c88a6eb40336eefdca4622c08 WatchSource:0}: Error finding container f1546d060823a1dc89c2d22a70d0d04a6a5bbb4c88a6eb40336eefdca4622c08: Status 404 returned error can't find the container with id f1546d060823a1dc89c2d22a70d0d04a6a5bbb4c88a6eb40336eefdca4622c08
Sep 14 10:13:00.778089 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:00.778036    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-51aefd/rate-test-model-51aefd-kserve-5d5f889dc5-949bj" event={"ID":"209ae11a-204c-48ff-83ef-10f4c073290f","Type":"ContainerStarted","Data":"8dee28fc832854753e9d54b3d54ff9fcc97472c6f1769893f472203419f136a4"}
Sep 14 10:13:00.778482 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:00.778099    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-51aefd/rate-test-model-51aefd-kserve-5d5f889dc5-949bj" event={"ID":"209ae11a-204c-48ff-83ef-10f4c073290f","Type":"ContainerStarted","Data":"f1546d060823a1dc89c2d22a70d0d04a6a5bbb4c88a6eb40336eefdca4622c08"}
Sep 14 10:13:00.795452 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:00.795389    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-51aefd/rate-test-model-51aefd-kserve-5d5f889dc5-949bj" podStartSLOduration=1.795368668 podStartE2EDuration="1.795368668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:12:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:13:00.792695624 +0000 UTC m=+1769.832370246" watchObservedRunningTime="2026-09-14 10:13:00.795368668 +0000 UTC m=+1769.835043289"
Sep 14 10:13:03.694179 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:03.694108    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:13:03.790420 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:03.790377    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:13:08.504954 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:08.504881    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-51aefd/sub-test-model-51aefd-kserve-5b4f744b4c-cjs5l" podUID="5f0c7af7-e9e0-416f-b907-9edb230a9e48" containerName="main" containerID="cri-o://2e596f2483cdd4a18a0dfe9ba4683bcb4bd55cbb5be7e8d37d92e2242896224d" gracePeriod=45
Sep 14 10:13:08.769776 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:08.769694    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-51aefd/sub-test-model-51aefd-kserve-5b4f744b4c-cjs5l"
Sep 14 10:13:08.828696 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:08.828661    2563 generic.go:356] "Generic (PLEG): container finished" podID="5f0c7af7-e9e0-416f-b907-9edb230a9e48" containerID="2e596f2483cdd4a18a0dfe9ba4683bcb4bd55cbb5be7e8d37d92e2242896224d" exitCode=0
Sep 14 10:13:08.828919 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:08.828715    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-51aefd/sub-test-model-51aefd-kserve-5b4f744b4c-cjs5l"
Sep 14 10:13:08.828919 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:08.828761    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-51aefd/sub-test-model-51aefd-kserve-5b4f744b4c-cjs5l" event={"ID":"5f0c7af7-e9e0-416f-b907-9edb230a9e48","Type":"ContainerDied","Data":"2e596f2483cdd4a18a0dfe9ba4683bcb4bd55cbb5be7e8d37d92e2242896224d"}
Sep 14 10:13:08.828919 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:08.828813    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-51aefd/sub-test-model-51aefd-kserve-5b4f744b4c-cjs5l" event={"ID":"5f0c7af7-e9e0-416f-b907-9edb230a9e48","Type":"ContainerDied","Data":"1732f5c9ae8da542df91eb6b31199b09b871ef372bc07f150c0d0833f2ac4e5d"}
Sep 14 10:13:08.828919 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:08.828835    2563 scope.go:117] "RemoveContainer" containerID="2e596f2483cdd4a18a0dfe9ba4683bcb4bd55cbb5be7e8d37d92e2242896224d"
Sep 14 10:13:08.838882 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:08.838859    2563 scope.go:117] "RemoveContainer" containerID="2e596f2483cdd4a18a0dfe9ba4683bcb4bd55cbb5be7e8d37d92e2242896224d"
Sep 14 10:13:08.839273 ip-10-0-132-4 kubenswrapper[2563]: E0914 10:13:08.839250    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e596f2483cdd4a18a0dfe9ba4683bcb4bd55cbb5be7e8d37d92e2242896224d\": container with ID starting with 2e596f2483cdd4a18a0dfe9ba4683bcb4bd55cbb5be7e8d37d92e2242896224d not found: ID does not exist" containerID="2e596f2483cdd4a18a0dfe9ba4683bcb4bd55cbb5be7e8d37d92e2242896224d"
Sep 14 10:13:08.839335 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:08.839278    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e596f2483cdd4a18a0dfe9ba4683bcb4bd55cbb5be7e8d37d92e2242896224d"} err="failed to get container status \"2e596f2483cdd4a18a0dfe9ba4683bcb4bd55cbb5be7e8d37d92e2242896224d\": rpc error: code = NotFound desc = could not find container \"2e596f2483cdd4a18a0dfe9ba4683bcb4bd55cbb5be7e8d37d92e2242896224d\": container with ID starting with 2e596f2483cdd4a18a0dfe9ba4683bcb4bd55cbb5be7e8d37d92e2242896224d not found: ID does not exist"
Sep 14 10:13:08.954930 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:08.954889    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f0c7af7-e9e0-416f-b907-9edb230a9e48-dshm\") pod \"5f0c7af7-e9e0-416f-b907-9edb230a9e48\" (UID: \"5f0c7af7-e9e0-416f-b907-9edb230a9e48\") "
Sep 14 10:13:08.955132 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:08.954963    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f0c7af7-e9e0-416f-b907-9edb230a9e48-tmp-dir\") pod \"5f0c7af7-e9e0-416f-b907-9edb230a9e48\" (UID: \"5f0c7af7-e9e0-416f-b907-9edb230a9e48\") "
Sep 14 10:13:08.955132 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:08.954990    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rx5nj\" (UniqueName: \"kubernetes.io/projected/5f0c7af7-e9e0-416f-b907-9edb230a9e48-kube-api-access-rx5nj\") pod \"5f0c7af7-e9e0-416f-b907-9edb230a9e48\" (UID: \"5f0c7af7-e9e0-416f-b907-9edb230a9e48\") "
Sep 14 10:13:08.955132 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:08.955045    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f0c7af7-e9e0-416f-b907-9edb230a9e48-model-cache\") pod \"5f0c7af7-e9e0-416f-b907-9edb230a9e48\" (UID: \"5f0c7af7-e9e0-416f-b907-9edb230a9e48\") "
Sep 14 10:13:08.955313 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:08.955183    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f0c7af7-e9e0-416f-b907-9edb230a9e48-tls-certs\") pod \"5f0c7af7-e9e0-416f-b907-9edb230a9e48\" (UID: \"5f0c7af7-e9e0-416f-b907-9edb230a9e48\") "
Sep 14 10:13:08.955313 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:08.955233    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f0c7af7-e9e0-416f-b907-9edb230a9e48-home\") pod \"5f0c7af7-e9e0-416f-b907-9edb230a9e48\" (UID: \"5f0c7af7-e9e0-416f-b907-9edb230a9e48\") "
Sep 14 10:13:08.955313 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:08.955283    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f0c7af7-e9e0-416f-b907-9edb230a9e48-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5f0c7af7-e9e0-416f-b907-9edb230a9e48" (UID: "5f0c7af7-e9e0-416f-b907-9edb230a9e48"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:13:08.955313 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:08.955292    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f0c7af7-e9e0-416f-b907-9edb230a9e48-model-cache" (OuterVolumeSpecName: "model-cache") pod "5f0c7af7-e9e0-416f-b907-9edb230a9e48" (UID: "5f0c7af7-e9e0-416f-b907-9edb230a9e48"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:13:08.955533 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:08.955493    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f0c7af7-e9e0-416f-b907-9edb230a9e48-home" (OuterVolumeSpecName: "home") pod "5f0c7af7-e9e0-416f-b907-9edb230a9e48" (UID: "5f0c7af7-e9e0-416f-b907-9edb230a9e48"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:13:08.955625 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:08.955610    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f0c7af7-e9e0-416f-b907-9edb230a9e48-tmp-dir\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:13:08.955663 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:08.955627    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f0c7af7-e9e0-416f-b907-9edb230a9e48-model-cache\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:13:08.955663 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:08.955637    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f0c7af7-e9e0-416f-b907-9edb230a9e48-home\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:13:08.957260 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:08.957221    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f0c7af7-e9e0-416f-b907-9edb230a9e48-dshm" (OuterVolumeSpecName: "dshm") pod "5f0c7af7-e9e0-416f-b907-9edb230a9e48" (UID: "5f0c7af7-e9e0-416f-b907-9edb230a9e48"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:13:08.957396 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:08.957260    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f0c7af7-e9e0-416f-b907-9edb230a9e48-kube-api-access-rx5nj" (OuterVolumeSpecName: "kube-api-access-rx5nj") pod "5f0c7af7-e9e0-416f-b907-9edb230a9e48" (UID: "5f0c7af7-e9e0-416f-b907-9edb230a9e48"). InnerVolumeSpecName "kube-api-access-rx5nj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:13:08.957396 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:08.957282    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f0c7af7-e9e0-416f-b907-9edb230a9e48-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5f0c7af7-e9e0-416f-b907-9edb230a9e48" (UID: "5f0c7af7-e9e0-416f-b907-9edb230a9e48"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:13:09.056553 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:09.056445    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rx5nj\" (UniqueName: \"kubernetes.io/projected/5f0c7af7-e9e0-416f-b907-9edb230a9e48-kube-api-access-rx5nj\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:13:09.056553 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:09.056483    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f0c7af7-e9e0-416f-b907-9edb230a9e48-tls-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:13:09.056553 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:09.056498    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f0c7af7-e9e0-416f-b907-9edb230a9e48-dshm\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:13:09.150904 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:09.150862    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-51aefd/sub-test-model-51aefd-kserve-5b4f744b4c-cjs5l"]
Sep 14 10:13:09.154155 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:09.154111    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-51aefd/sub-test-model-51aefd-kserve-5b4f744b4c-cjs5l"]
Sep 14 10:13:09.390056 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:09.389926    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-f4d054/sub-test-model-f4d054-kserve-5cc6ff456f-vzc48" podUID="6e128dc2-1f25-4d94-8ff2-943b1d6338bc" containerName="main" containerID="cri-o://a82895699de9f19d8d7009d494df095684654ec7dd0f217fd52787653dd46572" gracePeriod=45
Sep 14 10:13:09.482612 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:09.482571    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f0c7af7-e9e0-416f-b907-9edb230a9e48" path="/var/lib/kubelet/pods/5f0c7af7-e9e0-416f-b907-9edb230a9e48/volumes"
Sep 14 10:13:09.656654 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:09.656617    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f4d054/sub-test-model-f4d054-kserve-5cc6ff456f-vzc48"
Sep 14 10:13:09.762107 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:09.762062    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e128dc2-1f25-4d94-8ff2-943b1d6338bc-tls-certs\") pod \"6e128dc2-1f25-4d94-8ff2-943b1d6338bc\" (UID: \"6e128dc2-1f25-4d94-8ff2-943b1d6338bc\") "
Sep 14 10:13:09.762289 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:09.762155    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e128dc2-1f25-4d94-8ff2-943b1d6338bc-home\") pod \"6e128dc2-1f25-4d94-8ff2-943b1d6338bc\" (UID: \"6e128dc2-1f25-4d94-8ff2-943b1d6338bc\") "
Sep 14 10:13:09.762289 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:09.762205    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xfcjb\" (UniqueName: \"kubernetes.io/projected/6e128dc2-1f25-4d94-8ff2-943b1d6338bc-kube-api-access-xfcjb\") pod \"6e128dc2-1f25-4d94-8ff2-943b1d6338bc\" (UID: \"6e128dc2-1f25-4d94-8ff2-943b1d6338bc\") "
Sep 14 10:13:09.762289 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:09.762244    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e128dc2-1f25-4d94-8ff2-943b1d6338bc-model-cache\") pod \"6e128dc2-1f25-4d94-8ff2-943b1d6338bc\" (UID: \"6e128dc2-1f25-4d94-8ff2-943b1d6338bc\") "
Sep 14 10:13:09.762407 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:09.762325    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e128dc2-1f25-4d94-8ff2-943b1d6338bc-dshm\") pod \"6e128dc2-1f25-4d94-8ff2-943b1d6338bc\" (UID: \"6e128dc2-1f25-4d94-8ff2-943b1d6338bc\") "
Sep 14 10:13:09.762407 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:09.762350    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e128dc2-1f25-4d94-8ff2-943b1d6338bc-tmp-dir\") pod \"6e128dc2-1f25-4d94-8ff2-943b1d6338bc\" (UID: \"6e128dc2-1f25-4d94-8ff2-943b1d6338bc\") "
Sep 14 10:13:09.762495 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:09.762442    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e128dc2-1f25-4d94-8ff2-943b1d6338bc-home" (OuterVolumeSpecName: "home") pod "6e128dc2-1f25-4d94-8ff2-943b1d6338bc" (UID: "6e128dc2-1f25-4d94-8ff2-943b1d6338bc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:13:09.762584 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:09.762561    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e128dc2-1f25-4d94-8ff2-943b1d6338bc-model-cache" (OuterVolumeSpecName: "model-cache") pod "6e128dc2-1f25-4d94-8ff2-943b1d6338bc" (UID: "6e128dc2-1f25-4d94-8ff2-943b1d6338bc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:13:09.762722 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:09.762698    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e128dc2-1f25-4d94-8ff2-943b1d6338bc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6e128dc2-1f25-4d94-8ff2-943b1d6338bc" (UID: "6e128dc2-1f25-4d94-8ff2-943b1d6338bc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:13:09.762788 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:09.762726    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e128dc2-1f25-4d94-8ff2-943b1d6338bc-home\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:13:09.762788 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:09.762741    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e128dc2-1f25-4d94-8ff2-943b1d6338bc-model-cache\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:13:09.764419 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:09.764374    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e128dc2-1f25-4d94-8ff2-943b1d6338bc-dshm" (OuterVolumeSpecName: "dshm") pod "6e128dc2-1f25-4d94-8ff2-943b1d6338bc" (UID: "6e128dc2-1f25-4d94-8ff2-943b1d6338bc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:13:09.764419 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:09.764402    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e128dc2-1f25-4d94-8ff2-943b1d6338bc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6e128dc2-1f25-4d94-8ff2-943b1d6338bc" (UID: "6e128dc2-1f25-4d94-8ff2-943b1d6338bc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:13:09.764589 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:09.764438    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e128dc2-1f25-4d94-8ff2-943b1d6338bc-kube-api-access-xfcjb" (OuterVolumeSpecName: "kube-api-access-xfcjb") pod "6e128dc2-1f25-4d94-8ff2-943b1d6338bc" (UID: "6e128dc2-1f25-4d94-8ff2-943b1d6338bc"). InnerVolumeSpecName "kube-api-access-xfcjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:13:09.837620 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:09.837585    2563 generic.go:356] "Generic (PLEG): container finished" podID="6e128dc2-1f25-4d94-8ff2-943b1d6338bc" containerID="a82895699de9f19d8d7009d494df095684654ec7dd0f217fd52787653dd46572" exitCode=0
Sep 14 10:13:09.837798 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:09.837637    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f4d054/sub-test-model-f4d054-kserve-5cc6ff456f-vzc48"
Sep 14 10:13:09.837798 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:09.837671    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f4d054/sub-test-model-f4d054-kserve-5cc6ff456f-vzc48" event={"ID":"6e128dc2-1f25-4d94-8ff2-943b1d6338bc","Type":"ContainerDied","Data":"a82895699de9f19d8d7009d494df095684654ec7dd0f217fd52787653dd46572"}
Sep 14 10:13:09.837798 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:09.837712    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f4d054/sub-test-model-f4d054-kserve-5cc6ff456f-vzc48" event={"ID":"6e128dc2-1f25-4d94-8ff2-943b1d6338bc","Type":"ContainerDied","Data":"5f4d5675b175b4d21658495cde7bdec14323e50b8ecdcb7ee959a893e7d60818"}
Sep 14 10:13:09.837798 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:09.837729    2563 scope.go:117] "RemoveContainer" containerID="a82895699de9f19d8d7009d494df095684654ec7dd0f217fd52787653dd46572"
Sep 14 10:13:09.847906 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:09.847879    2563 scope.go:117] "RemoveContainer" containerID="a82895699de9f19d8d7009d494df095684654ec7dd0f217fd52787653dd46572"
Sep 14 10:13:09.848343 ip-10-0-132-4 kubenswrapper[2563]: E0914 10:13:09.848314    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a82895699de9f19d8d7009d494df095684654ec7dd0f217fd52787653dd46572\": container with ID starting with a82895699de9f19d8d7009d494df095684654ec7dd0f217fd52787653dd46572 not found: ID does not exist" containerID="a82895699de9f19d8d7009d494df095684654ec7dd0f217fd52787653dd46572"
Sep 14 10:13:09.848441 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:09.848352    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a82895699de9f19d8d7009d494df095684654ec7dd0f217fd52787653dd46572"} err="failed to get container status \"a82895699de9f19d8d7009d494df095684654ec7dd0f217fd52787653dd46572\": rpc error: code = NotFound desc = could not find container \"a82895699de9f19d8d7009d494df095684654ec7dd0f217fd52787653dd46572\": container with ID starting with a82895699de9f19d8d7009d494df095684654ec7dd0f217fd52787653dd46572 not found: ID does not exist"
Sep 14 10:13:09.858889 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:09.858847    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f4d054/sub-test-model-f4d054-kserve-5cc6ff456f-vzc48"]
Sep 14 10:13:09.862240 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:09.862203    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f4d054/sub-test-model-f4d054-kserve-5cc6ff456f-vzc48"]
Sep 14 10:13:09.863894 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:09.863869    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e128dc2-1f25-4d94-8ff2-943b1d6338bc-dshm\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:13:09.863958 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:09.863896    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e128dc2-1f25-4d94-8ff2-943b1d6338bc-tmp-dir\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:13:09.863958 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:09.863906    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e128dc2-1f25-4d94-8ff2-943b1d6338bc-tls-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:13:09.863958 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:09.863915    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfcjb\" (UniqueName: \"kubernetes.io/projected/6e128dc2-1f25-4d94-8ff2-943b1d6338bc-kube-api-access-xfcjb\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:13:10.078287 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:10.078243    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-51aefd/rate-test-model-51aefd-kserve-5d5f889dc5-949bj"
Sep 14 10:13:10.078495 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:10.078305    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-51aefd/rate-test-model-51aefd-kserve-5d5f889dc5-949bj"
Sep 14 10:13:10.091648 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:10.091613    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-51aefd/rate-test-model-51aefd-kserve-5d5f889dc5-949bj"
Sep 14 10:13:10.857259 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:10.857228    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-51aefd/rate-test-model-51aefd-kserve-5d5f889dc5-949bj"
Sep 14 10:13:11.483914 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:11.483875    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e128dc2-1f25-4d94-8ff2-943b1d6338bc" path="/var/lib/kubelet/pods/6e128dc2-1f25-4d94-8ff2-943b1d6338bc/volumes"
Sep 14 10:13:22.158359 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:22.158314    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-f4d054/rate-test-model-f4d054-kserve-c6b747dc5-65ds2"]
Sep 14 10:13:22.158815 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:22.158773    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e128dc2-1f25-4d94-8ff2-943b1d6338bc" containerName="main"
Sep 14 10:13:22.158815 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:22.158788    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e128dc2-1f25-4d94-8ff2-943b1d6338bc" containerName="main"
Sep 14 10:13:22.158815 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:22.158809    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f0c7af7-e9e0-416f-b907-9edb230a9e48" containerName="main"
Sep 14 10:13:22.158815 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:22.158815    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f0c7af7-e9e0-416f-b907-9edb230a9e48" containerName="main"
Sep 14 10:13:22.158971 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:22.158872    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f0c7af7-e9e0-416f-b907-9edb230a9e48" containerName="main"
Sep 14 10:13:22.158971 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:22.158880    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e128dc2-1f25-4d94-8ff2-943b1d6338bc" containerName="main"
Sep 14 10:13:22.163236 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:22.163206    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f4d054/rate-test-model-f4d054-kserve-c6b747dc5-65ds2"
Sep 14 10:13:22.165860 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:22.165825    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-f4d054\"/\"rate-test-model-f4d054-kserve-self-signed-certs\""
Sep 14 10:13:22.165860 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:22.165825    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-f4d054\"/\"default-dockercfg-wnn54\""
Sep 14 10:13:22.166083 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:22.165825    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-f4d054\"/\"kube-root-ca.crt\""
Sep 14 10:13:22.166083 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:22.165839    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-f4d054\"/\"openshift-service-ca.crt\""
Sep 14 10:13:22.173184 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:22.173153    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f4d054/rate-test-model-f4d054-kserve-c6b747dc5-65ds2"]
Sep 14 10:13:22.267950 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:22.267907    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjqmt\" (UniqueName: \"kubernetes.io/projected/44b769c5-e3a2-4882-8d08-1ceb06719407-kube-api-access-gjqmt\") pod \"rate-test-model-f4d054-kserve-c6b747dc5-65ds2\" (UID: \"44b769c5-e3a2-4882-8d08-1ceb06719407\") " pod="ai-tenant-e2e-shared-b-w3-f4d054/rate-test-model-f4d054-kserve-c6b747dc5-65ds2"
Sep 14 10:13:22.268170 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:22.267994    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44b769c5-e3a2-4882-8d08-1ceb06719407-model-cache\") pod \"rate-test-model-f4d054-kserve-c6b747dc5-65ds2\" (UID: \"44b769c5-e3a2-4882-8d08-1ceb06719407\") " pod="ai-tenant-e2e-shared-b-w3-f4d054/rate-test-model-f4d054-kserve-c6b747dc5-65ds2"
Sep 14 10:13:22.268170 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:22.268058    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44b769c5-e3a2-4882-8d08-1ceb06719407-tls-certs\") pod \"rate-test-model-f4d054-kserve-c6b747dc5-65ds2\" (UID: \"44b769c5-e3a2-4882-8d08-1ceb06719407\") " pod="ai-tenant-e2e-shared-b-w3-f4d054/rate-test-model-f4d054-kserve-c6b747dc5-65ds2"
Sep 14 10:13:22.268170 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:22.268115    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44b769c5-e3a2-4882-8d08-1ceb06719407-home\") pod \"rate-test-model-f4d054-kserve-c6b747dc5-65ds2\" (UID: \"44b769c5-e3a2-4882-8d08-1ceb06719407\") " pod="ai-tenant-e2e-shared-b-w3-f4d054/rate-test-model-f4d054-kserve-c6b747dc5-65ds2"
Sep 14 10:13:22.268170 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:22.268145    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44b769c5-e3a2-4882-8d08-1ceb06719407-dshm\") pod \"rate-test-model-f4d054-kserve-c6b747dc5-65ds2\" (UID: \"44b769c5-e3a2-4882-8d08-1ceb06719407\") " pod="ai-tenant-e2e-shared-b-w3-f4d054/rate-test-model-f4d054-kserve-c6b747dc5-65ds2"
Sep 14 10:13:22.268339 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:22.268233    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44b769c5-e3a2-4882-8d08-1ceb06719407-tmp-dir\") pod \"rate-test-model-f4d054-kserve-c6b747dc5-65ds2\" (UID: \"44b769c5-e3a2-4882-8d08-1ceb06719407\") " pod="ai-tenant-e2e-shared-b-w3-f4d054/rate-test-model-f4d054-kserve-c6b747dc5-65ds2"
Sep 14 10:13:22.367894 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:22.367858    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:13:22.369061 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:22.369026    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44b769c5-e3a2-4882-8d08-1ceb06719407-tls-certs\") pod \"rate-test-model-f4d054-kserve-c6b747dc5-65ds2\" (UID: \"44b769c5-e3a2-4882-8d08-1ceb06719407\") " pod="ai-tenant-e2e-shared-b-w3-f4d054/rate-test-model-f4d054-kserve-c6b747dc5-65ds2"
Sep 14 10:13:22.369272 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:22.369093    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44b769c5-e3a2-4882-8d08-1ceb06719407-home\") pod \"rate-test-model-f4d054-kserve-c6b747dc5-65ds2\" (UID: \"44b769c5-e3a2-4882-8d08-1ceb06719407\") " pod="ai-tenant-e2e-shared-b-w3-f4d054/rate-test-model-f4d054-kserve-c6b747dc5-65ds2"
Sep 14 10:13:22.369272 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:22.369126    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44b769c5-e3a2-4882-8d08-1ceb06719407-dshm\") pod \"rate-test-model-f4d054-kserve-c6b747dc5-65ds2\" (UID: \"44b769c5-e3a2-4882-8d08-1ceb06719407\") " pod="ai-tenant-e2e-shared-b-w3-f4d054/rate-test-model-f4d054-kserve-c6b747dc5-65ds2"
Sep 14 10:13:22.369272 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:22.369187    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44b769c5-e3a2-4882-8d08-1ceb06719407-tmp-dir\") pod \"rate-test-model-f4d054-kserve-c6b747dc5-65ds2\" (UID: \"44b769c5-e3a2-4882-8d08-1ceb06719407\") " pod="ai-tenant-e2e-shared-b-w3-f4d054/rate-test-model-f4d054-kserve-c6b747dc5-65ds2"
Sep 14 10:13:22.369272 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:22.369265    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjqmt\" (UniqueName: \"kubernetes.io/projected/44b769c5-e3a2-4882-8d08-1ceb06719407-kube-api-access-gjqmt\") pod \"rate-test-model-f4d054-kserve-c6b747dc5-65ds2\" (UID: \"44b769c5-e3a2-4882-8d08-1ceb06719407\") " pod="ai-tenant-e2e-shared-b-w3-f4d054/rate-test-model-f4d054-kserve-c6b747dc5-65ds2"
Sep 14 10:13:22.369534 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:22.369312    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44b769c5-e3a2-4882-8d08-1ceb06719407-model-cache\") pod \"rate-test-model-f4d054-kserve-c6b747dc5-65ds2\" (UID: \"44b769c5-e3a2-4882-8d08-1ceb06719407\") " pod="ai-tenant-e2e-shared-b-w3-f4d054/rate-test-model-f4d054-kserve-c6b747dc5-65ds2"
Sep 14 10:13:22.369833 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:22.369809    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44b769c5-e3a2-4882-8d08-1ceb06719407-home\") pod \"rate-test-model-f4d054-kserve-c6b747dc5-65ds2\" (UID: \"44b769c5-e3a2-4882-8d08-1ceb06719407\") " pod="ai-tenant-e2e-shared-b-w3-f4d054/rate-test-model-f4d054-kserve-c6b747dc5-65ds2"
Sep 14 10:13:22.370026 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:22.369828    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44b769c5-e3a2-4882-8d08-1ceb06719407-tmp-dir\") pod \"rate-test-model-f4d054-kserve-c6b747dc5-65ds2\" (UID: \"44b769c5-e3a2-4882-8d08-1ceb06719407\") " pod="ai-tenant-e2e-shared-b-w3-f4d054/rate-test-model-f4d054-kserve-c6b747dc5-65ds2"
Sep 14 10:13:22.370442 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:22.370411    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44b769c5-e3a2-4882-8d08-1ceb06719407-model-cache\") pod \"rate-test-model-f4d054-kserve-c6b747dc5-65ds2\" (UID: \"44b769c5-e3a2-4882-8d08-1ceb06719407\") " pod="ai-tenant-e2e-shared-b-w3-f4d054/rate-test-model-f4d054-kserve-c6b747dc5-65ds2"
Sep 14 10:13:22.372656 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:22.372620    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44b769c5-e3a2-4882-8d08-1ceb06719407-dshm\") pod \"rate-test-model-f4d054-kserve-c6b747dc5-65ds2\" (UID: \"44b769c5-e3a2-4882-8d08-1ceb06719407\") " pod="ai-tenant-e2e-shared-b-w3-f4d054/rate-test-model-f4d054-kserve-c6b747dc5-65ds2"
Sep 14 10:13:22.372773 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:22.372718    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44b769c5-e3a2-4882-8d08-1ceb06719407-tls-certs\") pod \"rate-test-model-f4d054-kserve-c6b747dc5-65ds2\" (UID: \"44b769c5-e3a2-4882-8d08-1ceb06719407\") " pod="ai-tenant-e2e-shared-b-w3-f4d054/rate-test-model-f4d054-kserve-c6b747dc5-65ds2"
Sep 14 10:13:22.378103 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:22.378077    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjqmt\" (UniqueName: \"kubernetes.io/projected/44b769c5-e3a2-4882-8d08-1ceb06719407-kube-api-access-gjqmt\") pod \"rate-test-model-f4d054-kserve-c6b747dc5-65ds2\" (UID: \"44b769c5-e3a2-4882-8d08-1ceb06719407\") " pod="ai-tenant-e2e-shared-b-w3-f4d054/rate-test-model-f4d054-kserve-c6b747dc5-65ds2"
Sep 14 10:13:22.473953 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:22.473846    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f4d054/rate-test-model-f4d054-kserve-c6b747dc5-65ds2"
Sep 14 10:13:22.756358 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:22.756328    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f4d054/rate-test-model-f4d054-kserve-c6b747dc5-65ds2"]
Sep 14 10:13:22.758176 ip-10-0-132-4 kubenswrapper[2563]: W0914 10:13:22.758132    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod44b769c5_e3a2_4882_8d08_1ceb06719407.slice/crio-293c870f2ea87c211d4efaea9ddc0a1412f7961cf164025e935fb1ea2e6c2f83 WatchSource:0}: Error finding container 293c870f2ea87c211d4efaea9ddc0a1412f7961cf164025e935fb1ea2e6c2f83: Status 404 returned error can't find the container with id 293c870f2ea87c211d4efaea9ddc0a1412f7961cf164025e935fb1ea2e6c2f83
Sep 14 10:13:22.923015 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:22.922954    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f4d054/rate-test-model-f4d054-kserve-c6b747dc5-65ds2" event={"ID":"44b769c5-e3a2-4882-8d08-1ceb06719407","Type":"ContainerStarted","Data":"ec172bce5af1af3b26a5020c69f4f47c146d7e307666db5e960d6ed9f725a3d6"}
Sep 14 10:13:22.923224 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:22.923036    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f4d054/rate-test-model-f4d054-kserve-c6b747dc5-65ds2" event={"ID":"44b769c5-e3a2-4882-8d08-1ceb06719407","Type":"ContainerStarted","Data":"293c870f2ea87c211d4efaea9ddc0a1412f7961cf164025e935fb1ea2e6c2f83"}
Sep 14 10:13:22.940339 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:22.940282    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-f4d054/rate-test-model-f4d054-kserve-c6b747dc5-65ds2" podStartSLOduration=0.940264116 podStartE2EDuration="940.264116ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:13:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:13:22.939104247 +0000 UTC m=+1791.978778871" watchObservedRunningTime="2026-09-14 10:13:22.940264116 +0000 UTC m=+1791.979938755"
Sep 14 10:13:31.997331 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:31.997283    2563 scope.go:117] "RemoveContainer" containerID="de66c10475da8f46b0d3406b641f44a0cec688d223d509e328db1555359797e4"
Sep 14 10:13:32.474563 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:32.474475    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-f4d054/rate-test-model-f4d054-kserve-c6b747dc5-65ds2"
Sep 14 10:13:32.474563 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:32.474536    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-f4d054/rate-test-model-f4d054-kserve-c6b747dc5-65ds2"
Sep 14 10:13:32.487726 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:32.487689    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-f4d054/rate-test-model-f4d054-kserve-c6b747dc5-65ds2"
Sep 14 10:13:32.996152 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:32.996122    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-f4d054/rate-test-model-f4d054-kserve-c6b747dc5-65ds2"
Sep 14 10:13:54.675051 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:54.674960    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:13:57.445335 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:57.445295    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-51aefd/rate-test-model-51aefd-kserve-5d5f889dc5-949bj"]
Sep 14 10:13:57.463385 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:57.463344    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:13:58.548958 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:58.548916    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f4d054/rate-test-model-f4d054-kserve-c6b747dc5-65ds2"]
Sep 14 10:13:58.565118 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:58.565056    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:13:58.974838 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:58.974703    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:13:59.264783 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:59.264743    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc"]
Sep 14 10:13:59.268386 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:59.268358    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc"
Sep 14 10:13:59.270570 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:59.270546    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-78742b-openshift-default-dockercfg-nw69m\""
Sep 14 10:13:59.281184 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:59.281145    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:13:59.282912 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:59.282880    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc"]
Sep 14 10:13:59.431672 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:59.431634    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:13:59.434908 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:59.434870    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/150274d4-9512-4493-8dd0-631bcb8ce385-istio-data\") pod \"e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc\" (UID: \"150274d4-9512-4493-8dd0-631bcb8ce385\") " pod="openshift-ingress/e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc"
Sep 14 10:13:59.435040 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:59.434927    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/150274d4-9512-4493-8dd0-631bcb8ce385-workload-certs\") pod \"e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc\" (UID: \"150274d4-9512-4493-8dd0-631bcb8ce385\") " pod="openshift-ingress/e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc"
Sep 14 10:13:59.435040 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:59.434953    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/150274d4-9512-4493-8dd0-631bcb8ce385-workload-socket\") pod \"e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc\" (UID: \"150274d4-9512-4493-8dd0-631bcb8ce385\") " pod="openshift-ingress/e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc"
Sep 14 10:13:59.435119 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:59.435080    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/150274d4-9512-4493-8dd0-631bcb8ce385-istio-podinfo\") pod \"e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc\" (UID: \"150274d4-9512-4493-8dd0-631bcb8ce385\") " pod="openshift-ingress/e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc"
Sep 14 10:13:59.435167 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:59.435114    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/150274d4-9512-4493-8dd0-631bcb8ce385-istiod-ca-cert\") pod \"e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc\" (UID: \"150274d4-9512-4493-8dd0-631bcb8ce385\") " pod="openshift-ingress/e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc"
Sep 14 10:13:59.435240 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:59.435214    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/150274d4-9512-4493-8dd0-631bcb8ce385-credential-socket\") pod \"e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc\" (UID: \"150274d4-9512-4493-8dd0-631bcb8ce385\") " pod="openshift-ingress/e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc"
Sep 14 10:13:59.435298 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:59.435264    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/150274d4-9512-4493-8dd0-631bcb8ce385-istio-token\") pod \"e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc\" (UID: \"150274d4-9512-4493-8dd0-631bcb8ce385\") " pod="openshift-ingress/e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc"
Sep 14 10:13:59.435353 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:59.435303    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/150274d4-9512-4493-8dd0-631bcb8ce385-istio-envoy\") pod \"e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc\" (UID: \"150274d4-9512-4493-8dd0-631bcb8ce385\") " pod="openshift-ingress/e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc"
Sep 14 10:13:59.435394 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:59.435361    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ds2d\" (UniqueName: \"kubernetes.io/projected/150274d4-9512-4493-8dd0-631bcb8ce385-kube-api-access-8ds2d\") pod \"e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc\" (UID: \"150274d4-9512-4493-8dd0-631bcb8ce385\") " pod="openshift-ingress/e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc"
Sep 14 10:13:59.536022 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:59.535904    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/150274d4-9512-4493-8dd0-631bcb8ce385-credential-socket\") pod \"e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc\" (UID: \"150274d4-9512-4493-8dd0-631bcb8ce385\") " pod="openshift-ingress/e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc"
Sep 14 10:13:59.536022 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:59.535953    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/150274d4-9512-4493-8dd0-631bcb8ce385-istio-token\") pod \"e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc\" (UID: \"150274d4-9512-4493-8dd0-631bcb8ce385\") " pod="openshift-ingress/e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc"
Sep 14 10:13:59.536022 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:59.535985    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/150274d4-9512-4493-8dd0-631bcb8ce385-istio-envoy\") pod \"e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc\" (UID: \"150274d4-9512-4493-8dd0-631bcb8ce385\") " pod="openshift-ingress/e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc"
Sep 14 10:13:59.536348 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:59.536051    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ds2d\" (UniqueName: \"kubernetes.io/projected/150274d4-9512-4493-8dd0-631bcb8ce385-kube-api-access-8ds2d\") pod \"e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc\" (UID: \"150274d4-9512-4493-8dd0-631bcb8ce385\") " pod="openshift-ingress/e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc"
Sep 14 10:13:59.536348 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:59.536147    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/150274d4-9512-4493-8dd0-631bcb8ce385-istio-data\") pod \"e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc\" (UID: \"150274d4-9512-4493-8dd0-631bcb8ce385\") " pod="openshift-ingress/e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc"
Sep 14 10:13:59.536348 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:59.536193    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/150274d4-9512-4493-8dd0-631bcb8ce385-workload-certs\") pod \"e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc\" (UID: \"150274d4-9512-4493-8dd0-631bcb8ce385\") " pod="openshift-ingress/e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc"
Sep 14 10:13:59.536348 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:59.536222    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/150274d4-9512-4493-8dd0-631bcb8ce385-workload-socket\") pod \"e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc\" (UID: \"150274d4-9512-4493-8dd0-631bcb8ce385\") " pod="openshift-ingress/e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc"
Sep 14 10:13:59.536348 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:59.536266    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/150274d4-9512-4493-8dd0-631bcb8ce385-istio-podinfo\") pod \"e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc\" (UID: \"150274d4-9512-4493-8dd0-631bcb8ce385\") " pod="openshift-ingress/e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc"
Sep 14 10:13:59.536348 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:59.536297    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/150274d4-9512-4493-8dd0-631bcb8ce385-istiod-ca-cert\") pod \"e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc\" (UID: \"150274d4-9512-4493-8dd0-631bcb8ce385\") " pod="openshift-ingress/e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc"
Sep 14 10:13:59.536653 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:59.536445    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/150274d4-9512-4493-8dd0-631bcb8ce385-credential-socket\") pod \"e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc\" (UID: \"150274d4-9512-4493-8dd0-631bcb8ce385\") " pod="openshift-ingress/e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc"
Sep 14 10:13:59.536653 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:59.536571    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/150274d4-9512-4493-8dd0-631bcb8ce385-istio-data\") pod \"e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc\" (UID: \"150274d4-9512-4493-8dd0-631bcb8ce385\") " pod="openshift-ingress/e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc"
Sep 14 10:13:59.536653 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:59.536616    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/150274d4-9512-4493-8dd0-631bcb8ce385-workload-certs\") pod \"e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc\" (UID: \"150274d4-9512-4493-8dd0-631bcb8ce385\") " pod="openshift-ingress/e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc"
Sep 14 10:13:59.537021 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:59.536653    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/150274d4-9512-4493-8dd0-631bcb8ce385-workload-socket\") pod \"e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc\" (UID: \"150274d4-9512-4493-8dd0-631bcb8ce385\") " pod="openshift-ingress/e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc"
Sep 14 10:13:59.537021 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:59.536908    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/150274d4-9512-4493-8dd0-631bcb8ce385-istiod-ca-cert\") pod \"e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc\" (UID: \"150274d4-9512-4493-8dd0-631bcb8ce385\") " pod="openshift-ingress/e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc"
Sep 14 10:13:59.538508 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:59.538478    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/150274d4-9512-4493-8dd0-631bcb8ce385-istio-envoy\") pod \"e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc\" (UID: \"150274d4-9512-4493-8dd0-631bcb8ce385\") " pod="openshift-ingress/e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc"
Sep 14 10:13:59.538821 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:59.538801    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/150274d4-9512-4493-8dd0-631bcb8ce385-istio-podinfo\") pod \"e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc\" (UID: \"150274d4-9512-4493-8dd0-631bcb8ce385\") " pod="openshift-ingress/e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc"
Sep 14 10:13:59.550151 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:59.550110    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/150274d4-9512-4493-8dd0-631bcb8ce385-istio-token\") pod \"e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc\" (UID: \"150274d4-9512-4493-8dd0-631bcb8ce385\") " pod="openshift-ingress/e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc"
Sep 14 10:13:59.550777 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:59.550753    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ds2d\" (UniqueName: \"kubernetes.io/projected/150274d4-9512-4493-8dd0-631bcb8ce385-kube-api-access-8ds2d\") pod \"e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc\" (UID: \"150274d4-9512-4493-8dd0-631bcb8ce385\") " pod="openshift-ingress/e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc"
Sep 14 10:13:59.580579 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:59.580534    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc"
Sep 14 10:13:59.865240 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:59.865199    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc"]
Sep 14 10:13:59.867800 ip-10-0-132-4 kubenswrapper[2563]: W0914 10:13:59.867768    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod150274d4_9512_4493_8dd0_631bcb8ce385.slice/crio-2fa53f4a74ccfe6615e482730cfb706ee4b00ea1fa707e7ef091361057484a04 WatchSource:0}: Error finding container 2fa53f4a74ccfe6615e482730cfb706ee4b00ea1fa707e7ef091361057484a04: Status 404 returned error can't find the container with id 2fa53f4a74ccfe6615e482730cfb706ee4b00ea1fa707e7ef091361057484a04
Sep 14 10:13:59.870235 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:59.870200    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 10:13:59.870332 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:59.870272    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 10:13:59.870332 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:13:59.870300    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 10:14:00.158931 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:00.157839    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc" event={"ID":"150274d4-9512-4493-8dd0-631bcb8ce385","Type":"ContainerStarted","Data":"ddc620850fc188aa493d9995584c4c58e711f0fb47bba7541cbf86787590383c"}
Sep 14 10:14:00.158931 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:00.157927    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc" event={"ID":"150274d4-9512-4493-8dd0-631bcb8ce385","Type":"ContainerStarted","Data":"2fa53f4a74ccfe6615e482730cfb706ee4b00ea1fa707e7ef091361057484a04"}
Sep 14 10:14:00.181749 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:00.181650    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc" podStartSLOduration=1.181621435 podStartE2EDuration="1.181621435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:13:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:14:00.177931331 +0000 UTC m=+1829.217605953" watchObservedRunningTime="2026-09-14 10:14:00.181621435 +0000 UTC m=+1829.221296062"
Sep 14 10:14:00.580674 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:00.580629    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc"
Sep 14 10:14:00.586223 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:00.586188    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc"
Sep 14 10:14:01.163620 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:01.163590    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc"
Sep 14 10:14:01.164710 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:01.164689    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc"
Sep 14 10:14:01.393366 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:01.393334    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:14:02.064459 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:02.064423    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:14:02.151789 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:02.151751    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:14:02.368720 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:02.368626    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:14:03.109350 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:03.109320    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:14:03.995066 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:03.995021    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:14:04.095217 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:04.095182    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:14:04.373915 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:04.373824    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:14:04.492419 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:04.492388    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:14:04.598899 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:04.598867    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:14:12.465768 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:12.465718    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-51aefd/rate-test-model-51aefd-kserve-5d5f889dc5-949bj" podUID="209ae11a-204c-48ff-83ef-10f4c073290f" containerName="main" containerID="cri-o://8dee28fc832854753e9d54b3d54ff9fcc97472c6f1769893f472203419f136a4" gracePeriod=45
Sep 14 10:14:12.755550 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:12.755509    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-51aefd/rate-test-model-51aefd-kserve-5d5f889dc5-949bj"
Sep 14 10:14:12.861572 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:12.861529    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/209ae11a-204c-48ff-83ef-10f4c073290f-home\") pod \"209ae11a-204c-48ff-83ef-10f4c073290f\" (UID: \"209ae11a-204c-48ff-83ef-10f4c073290f\") "
Sep 14 10:14:12.861572 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:12.861587    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mw6n5\" (UniqueName: \"kubernetes.io/projected/209ae11a-204c-48ff-83ef-10f4c073290f-kube-api-access-mw6n5\") pod \"209ae11a-204c-48ff-83ef-10f4c073290f\" (UID: \"209ae11a-204c-48ff-83ef-10f4c073290f\") "
Sep 14 10:14:12.861853 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:12.861628    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/209ae11a-204c-48ff-83ef-10f4c073290f-dshm\") pod \"209ae11a-204c-48ff-83ef-10f4c073290f\" (UID: \"209ae11a-204c-48ff-83ef-10f4c073290f\") "
Sep 14 10:14:12.861853 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:12.861649    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/209ae11a-204c-48ff-83ef-10f4c073290f-model-cache\") pod \"209ae11a-204c-48ff-83ef-10f4c073290f\" (UID: \"209ae11a-204c-48ff-83ef-10f4c073290f\") "
Sep 14 10:14:12.861853 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:12.861706    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/209ae11a-204c-48ff-83ef-10f4c073290f-tmp-dir\") pod \"209ae11a-204c-48ff-83ef-10f4c073290f\" (UID: \"209ae11a-204c-48ff-83ef-10f4c073290f\") "
Sep 14 10:14:12.861853 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:12.861740    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/209ae11a-204c-48ff-83ef-10f4c073290f-tls-certs\") pod \"209ae11a-204c-48ff-83ef-10f4c073290f\" (UID: \"209ae11a-204c-48ff-83ef-10f4c073290f\") "
Sep 14 10:14:12.862113 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:12.861850    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/209ae11a-204c-48ff-83ef-10f4c073290f-home" (OuterVolumeSpecName: "home") pod "209ae11a-204c-48ff-83ef-10f4c073290f" (UID: "209ae11a-204c-48ff-83ef-10f4c073290f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:14:12.862113 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:12.861946    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/209ae11a-204c-48ff-83ef-10f4c073290f-model-cache" (OuterVolumeSpecName: "model-cache") pod "209ae11a-204c-48ff-83ef-10f4c073290f" (UID: "209ae11a-204c-48ff-83ef-10f4c073290f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:14:12.862113 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:12.861979    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/209ae11a-204c-48ff-83ef-10f4c073290f-home\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:14:12.862113 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:12.861975    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/209ae11a-204c-48ff-83ef-10f4c073290f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "209ae11a-204c-48ff-83ef-10f4c073290f" (UID: "209ae11a-204c-48ff-83ef-10f4c073290f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:14:12.863894 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:12.863868    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/209ae11a-204c-48ff-83ef-10f4c073290f-dshm" (OuterVolumeSpecName: "dshm") pod "209ae11a-204c-48ff-83ef-10f4c073290f" (UID: "209ae11a-204c-48ff-83ef-10f4c073290f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:14:12.863995 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:12.863946    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/209ae11a-204c-48ff-83ef-10f4c073290f-kube-api-access-mw6n5" (OuterVolumeSpecName: "kube-api-access-mw6n5") pod "209ae11a-204c-48ff-83ef-10f4c073290f" (UID: "209ae11a-204c-48ff-83ef-10f4c073290f"). InnerVolumeSpecName "kube-api-access-mw6n5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:14:12.863995 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:12.863970    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/209ae11a-204c-48ff-83ef-10f4c073290f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "209ae11a-204c-48ff-83ef-10f4c073290f" (UID: "209ae11a-204c-48ff-83ef-10f4c073290f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:14:12.962535 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:12.962491    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/209ae11a-204c-48ff-83ef-10f4c073290f-dshm\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:14:12.962535 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:12.962525    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/209ae11a-204c-48ff-83ef-10f4c073290f-model-cache\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:14:12.962535 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:12.962537    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/209ae11a-204c-48ff-83ef-10f4c073290f-tmp-dir\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:14:12.962535 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:12.962547    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/209ae11a-204c-48ff-83ef-10f4c073290f-tls-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:14:12.962823 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:12.962557    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mw6n5\" (UniqueName: \"kubernetes.io/projected/209ae11a-204c-48ff-83ef-10f4c073290f-kube-api-access-mw6n5\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:14:13.242391 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:13.242187    2563 generic.go:356] "Generic (PLEG): container finished" podID="209ae11a-204c-48ff-83ef-10f4c073290f" containerID="8dee28fc832854753e9d54b3d54ff9fcc97472c6f1769893f472203419f136a4" exitCode=0
Sep 14 10:14:13.242391 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:13.242278    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-51aefd/rate-test-model-51aefd-kserve-5d5f889dc5-949bj" event={"ID":"209ae11a-204c-48ff-83ef-10f4c073290f","Type":"ContainerDied","Data":"8dee28fc832854753e9d54b3d54ff9fcc97472c6f1769893f472203419f136a4"}
Sep 14 10:14:13.242391 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:13.242348    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-51aefd/rate-test-model-51aefd-kserve-5d5f889dc5-949bj" event={"ID":"209ae11a-204c-48ff-83ef-10f4c073290f","Type":"ContainerDied","Data":"f1546d060823a1dc89c2d22a70d0d04a6a5bbb4c88a6eb40336eefdca4622c08"}
Sep 14 10:14:13.242391 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:13.242376    2563 scope.go:117] "RemoveContainer" containerID="8dee28fc832854753e9d54b3d54ff9fcc97472c6f1769893f472203419f136a4"
Sep 14 10:14:13.242803 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:13.242562    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-51aefd/rate-test-model-51aefd-kserve-5d5f889dc5-949bj"
Sep 14 10:14:13.257564 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:13.257541    2563 scope.go:117] "RemoveContainer" containerID="8dee28fc832854753e9d54b3d54ff9fcc97472c6f1769893f472203419f136a4"
Sep 14 10:14:13.257990 ip-10-0-132-4 kubenswrapper[2563]: E0914 10:14:13.257967    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dee28fc832854753e9d54b3d54ff9fcc97472c6f1769893f472203419f136a4\": container with ID starting with 8dee28fc832854753e9d54b3d54ff9fcc97472c6f1769893f472203419f136a4 not found: ID does not exist" containerID="8dee28fc832854753e9d54b3d54ff9fcc97472c6f1769893f472203419f136a4"
Sep 14 10:14:13.258096 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:13.258029    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dee28fc832854753e9d54b3d54ff9fcc97472c6f1769893f472203419f136a4"} err="failed to get container status \"8dee28fc832854753e9d54b3d54ff9fcc97472c6f1769893f472203419f136a4\": rpc error: code = NotFound desc = could not find container \"8dee28fc832854753e9d54b3d54ff9fcc97472c6f1769893f472203419f136a4\": container with ID starting with 8dee28fc832854753e9d54b3d54ff9fcc97472c6f1769893f472203419f136a4 not found: ID does not exist"
Sep 14 10:14:13.267781 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:13.267738    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-51aefd/rate-test-model-51aefd-kserve-5d5f889dc5-949bj"]
Sep 14 10:14:13.273177 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:13.273139    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-51aefd/rate-test-model-51aefd-kserve-5d5f889dc5-949bj"]
Sep 14 10:14:13.482563 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:13.482531    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="209ae11a-204c-48ff-83ef-10f4c073290f" path="/var/lib/kubelet/pods/209ae11a-204c-48ff-83ef-10f4c073290f/volumes"
Sep 14 10:14:13.568954 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:13.568864    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-f4d054/rate-test-model-f4d054-kserve-c6b747dc5-65ds2" podUID="44b769c5-e3a2-4882-8d08-1ceb06719407" containerName="main" containerID="cri-o://ec172bce5af1af3b26a5020c69f4f47c146d7e307666db5e960d6ed9f725a3d6" gracePeriod=45
Sep 14 10:14:13.857588 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:13.857552    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f4d054/rate-test-model-f4d054-kserve-c6b747dc5-65ds2"
Sep 14 10:14:13.971497 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:13.971459    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44b769c5-e3a2-4882-8d08-1ceb06719407-tmp-dir\") pod \"44b769c5-e3a2-4882-8d08-1ceb06719407\" (UID: \"44b769c5-e3a2-4882-8d08-1ceb06719407\") "
Sep 14 10:14:13.971681 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:13.971520    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gjqmt\" (UniqueName: \"kubernetes.io/projected/44b769c5-e3a2-4882-8d08-1ceb06719407-kube-api-access-gjqmt\") pod \"44b769c5-e3a2-4882-8d08-1ceb06719407\" (UID: \"44b769c5-e3a2-4882-8d08-1ceb06719407\") "
Sep 14 10:14:13.971681 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:13.971544    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44b769c5-e3a2-4882-8d08-1ceb06719407-dshm\") pod \"44b769c5-e3a2-4882-8d08-1ceb06719407\" (UID: \"44b769c5-e3a2-4882-8d08-1ceb06719407\") "
Sep 14 10:14:13.971681 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:13.971560    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44b769c5-e3a2-4882-8d08-1ceb06719407-home\") pod \"44b769c5-e3a2-4882-8d08-1ceb06719407\" (UID: \"44b769c5-e3a2-4882-8d08-1ceb06719407\") "
Sep 14 10:14:13.971681 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:13.971591    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44b769c5-e3a2-4882-8d08-1ceb06719407-model-cache\") pod \"44b769c5-e3a2-4882-8d08-1ceb06719407\" (UID: \"44b769c5-e3a2-4882-8d08-1ceb06719407\") "
Sep 14 10:14:13.971681 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:13.971655    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44b769c5-e3a2-4882-8d08-1ceb06719407-tls-certs\") pod \"44b769c5-e3a2-4882-8d08-1ceb06719407\" (UID: \"44b769c5-e3a2-4882-8d08-1ceb06719407\") "
Sep 14 10:14:13.971935 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:13.971770    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44b769c5-e3a2-4882-8d08-1ceb06719407-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "44b769c5-e3a2-4882-8d08-1ceb06719407" (UID: "44b769c5-e3a2-4882-8d08-1ceb06719407"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:14:13.971935 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:13.971853    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44b769c5-e3a2-4882-8d08-1ceb06719407-home" (OuterVolumeSpecName: "home") pod "44b769c5-e3a2-4882-8d08-1ceb06719407" (UID: "44b769c5-e3a2-4882-8d08-1ceb06719407"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:14:13.971935 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:13.971871    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44b769c5-e3a2-4882-8d08-1ceb06719407-model-cache" (OuterVolumeSpecName: "model-cache") pod "44b769c5-e3a2-4882-8d08-1ceb06719407" (UID: "44b769c5-e3a2-4882-8d08-1ceb06719407"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:14:13.972181 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:13.972036    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44b769c5-e3a2-4882-8d08-1ceb06719407-tmp-dir\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:14:13.972181 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:13.972056    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44b769c5-e3a2-4882-8d08-1ceb06719407-home\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:14:13.972181 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:13.972077    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44b769c5-e3a2-4882-8d08-1ceb06719407-model-cache\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:14:13.973756 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:13.973720    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44b769c5-e3a2-4882-8d08-1ceb06719407-dshm" (OuterVolumeSpecName: "dshm") pod "44b769c5-e3a2-4882-8d08-1ceb06719407" (UID: "44b769c5-e3a2-4882-8d08-1ceb06719407"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:14:13.973756 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:13.973734    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44b769c5-e3a2-4882-8d08-1ceb06719407-kube-api-access-gjqmt" (OuterVolumeSpecName: "kube-api-access-gjqmt") pod "44b769c5-e3a2-4882-8d08-1ceb06719407" (UID: "44b769c5-e3a2-4882-8d08-1ceb06719407"). InnerVolumeSpecName "kube-api-access-gjqmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:14:13.973925 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:13.973722    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44b769c5-e3a2-4882-8d08-1ceb06719407-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "44b769c5-e3a2-4882-8d08-1ceb06719407" (UID: "44b769c5-e3a2-4882-8d08-1ceb06719407"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:14:14.073523 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:14.073419    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44b769c5-e3a2-4882-8d08-1ceb06719407-tls-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:14:14.073523 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:14.073455    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjqmt\" (UniqueName: \"kubernetes.io/projected/44b769c5-e3a2-4882-8d08-1ceb06719407-kube-api-access-gjqmt\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:14:14.073523 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:14.073471    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44b769c5-e3a2-4882-8d08-1ceb06719407-dshm\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:14:14.261554 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:14.261514    2563 generic.go:356] "Generic (PLEG): container finished" podID="44b769c5-e3a2-4882-8d08-1ceb06719407" containerID="ec172bce5af1af3b26a5020c69f4f47c146d7e307666db5e960d6ed9f725a3d6" exitCode=0
Sep 14 10:14:14.261554 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:14.261560    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f4d054/rate-test-model-f4d054-kserve-c6b747dc5-65ds2" event={"ID":"44b769c5-e3a2-4882-8d08-1ceb06719407","Type":"ContainerDied","Data":"ec172bce5af1af3b26a5020c69f4f47c146d7e307666db5e960d6ed9f725a3d6"}
Sep 14 10:14:14.261806 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:14.261580    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f4d054/rate-test-model-f4d054-kserve-c6b747dc5-65ds2"
Sep 14 10:14:14.261806 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:14.261594    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f4d054/rate-test-model-f4d054-kserve-c6b747dc5-65ds2" event={"ID":"44b769c5-e3a2-4882-8d08-1ceb06719407","Type":"ContainerDied","Data":"293c870f2ea87c211d4efaea9ddc0a1412f7961cf164025e935fb1ea2e6c2f83"}
Sep 14 10:14:14.261806 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:14.261611    2563 scope.go:117] "RemoveContainer" containerID="ec172bce5af1af3b26a5020c69f4f47c146d7e307666db5e960d6ed9f725a3d6"
Sep 14 10:14:14.271747 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:14.271725    2563 scope.go:117] "RemoveContainer" containerID="ec172bce5af1af3b26a5020c69f4f47c146d7e307666db5e960d6ed9f725a3d6"
Sep 14 10:14:14.272203 ip-10-0-132-4 kubenswrapper[2563]: E0914 10:14:14.272176    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec172bce5af1af3b26a5020c69f4f47c146d7e307666db5e960d6ed9f725a3d6\": container with ID starting with ec172bce5af1af3b26a5020c69f4f47c146d7e307666db5e960d6ed9f725a3d6 not found: ID does not exist" containerID="ec172bce5af1af3b26a5020c69f4f47c146d7e307666db5e960d6ed9f725a3d6"
Sep 14 10:14:14.272265 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:14.272214    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec172bce5af1af3b26a5020c69f4f47c146d7e307666db5e960d6ed9f725a3d6"} err="failed to get container status \"ec172bce5af1af3b26a5020c69f4f47c146d7e307666db5e960d6ed9f725a3d6\": rpc error: code = NotFound desc = could not find container \"ec172bce5af1af3b26a5020c69f4f47c146d7e307666db5e960d6ed9f725a3d6\": container with ID starting with ec172bce5af1af3b26a5020c69f4f47c146d7e307666db5e960d6ed9f725a3d6 not found: ID does not exist"
Sep 14 10:14:14.283333 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:14.283292    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f4d054/rate-test-model-f4d054-kserve-c6b747dc5-65ds2"]
Sep 14 10:14:14.287097 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:14.287056    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f4d054/rate-test-model-f4d054-kserve-c6b747dc5-65ds2"]
Sep 14 10:14:15.482944 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:15.482896    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44b769c5-e3a2-4882-8d08-1ceb06719407" path="/var/lib/kubelet/pods/44b769c5-e3a2-4882-8d08-1ceb06719407/volumes"
Sep 14 10:14:17.206196 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:17.206148    2563 patch_prober.go:28] interesting pod/e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 14 10:14:17.206695 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:17.206216    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc" podUID="150274d4-9512-4493-8dd0-631bcb8ce385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 14 10:14:18.289620 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:18.289580    2563 generic.go:356] "Generic (PLEG): container finished" podID="150274d4-9512-4493-8dd0-631bcb8ce385" containerID="ddc620850fc188aa493d9995584c4c58e711f0fb47bba7541cbf86787590383c" exitCode=0
Sep 14 10:14:18.290089 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:18.289654    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc" event={"ID":"150274d4-9512-4493-8dd0-631bcb8ce385","Type":"ContainerDied","Data":"ddc620850fc188aa493d9995584c4c58e711f0fb47bba7541cbf86787590383c"}
Sep 14 10:14:18.290141 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:18.290128    2563 scope.go:117] "RemoveContainer" containerID="ddc620850fc188aa493d9995584c4c58e711f0fb47bba7541cbf86787590383c"
Sep 14 10:14:18.291307 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:18.291277    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 10:14:18.291373 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:18.291350    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 10:14:18.291409 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:18.291384    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 10:14:19.297464 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:19.297427    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc" event={"ID":"150274d4-9512-4493-8dd0-631bcb8ce385","Type":"ContainerStarted","Data":"571caf441d66f6813ca524ce93f37f44d23ddf00f747e7a968aa54c9356c7759"}
Sep 14 10:14:19.581228 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:19.581131    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc"
Sep 14 10:14:19.581399 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:19.581366    2563 patch_prober.go:28] interesting pod/e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Sep 14 10:14:19.581442 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:19.581417    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc" podUID="150274d4-9512-4493-8dd0-631bcb8ce385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Sep 14 10:14:20.581866 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:20.581818    2563 patch_prober.go:28] interesting pod/e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Sep 14 10:14:20.582324 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:20.581897    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc" podUID="150274d4-9512-4493-8dd0-631bcb8ce385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Sep 14 10:14:21.580938 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:21.580891    2563 patch_prober.go:28] interesting pod/e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Sep 14 10:14:21.581201 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:21.580957    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc" podUID="150274d4-9512-4493-8dd0-631bcb8ce385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Sep 14 10:14:22.581520 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:22.581477    2563 patch_prober.go:28] interesting pod/e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Sep 14 10:14:22.581932 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:22.581534    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc" podUID="150274d4-9512-4493-8dd0-631bcb8ce385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Sep 14 10:14:23.581144 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:23.581103    2563 patch_prober.go:28] interesting pod/e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Sep 14 10:14:23.581345 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:23.581177    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc" podUID="150274d4-9512-4493-8dd0-631bcb8ce385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Sep 14 10:14:24.581846 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:24.581798    2563 patch_prober.go:28] interesting pod/e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Sep 14 10:14:24.582313 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:24.581882    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc" podUID="150274d4-9512-4493-8dd0-631bcb8ce385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Sep 14 10:14:25.584968 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:25.584935    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc"
Sep 14 10:14:25.585394 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:25.585240    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc"
Sep 14 10:14:25.586173 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:25.586153    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc"
Sep 14 10:14:26.295792 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:26.295760    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:14:27.422493 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:27.422456    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:14:27.687355 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:27.687262    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:14:27.986367 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:27.986321    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n"]
Sep 14 10:14:27.986819 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:27.986798    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="209ae11a-204c-48ff-83ef-10f4c073290f" containerName="main"
Sep 14 10:14:27.986912 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:27.986821    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="209ae11a-204c-48ff-83ef-10f4c073290f" containerName="main"
Sep 14 10:14:27.986912 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:27.986855    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44b769c5-e3a2-4882-8d08-1ceb06719407" containerName="main"
Sep 14 10:14:27.986912 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:27.986865    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="44b769c5-e3a2-4882-8d08-1ceb06719407" containerName="main"
Sep 14 10:14:27.987103 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:27.986953    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="209ae11a-204c-48ff-83ef-10f4c073290f" containerName="main"
Sep 14 10:14:27.987103 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:27.986967    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="44b769c5-e3a2-4882-8d08-1ceb06719407" containerName="main"
Sep 14 10:14:27.989459 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:27.989433    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n"
Sep 14 10:14:27.992085 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:27.992049    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-2c2a39-openshift-default-dockercfg-j76f5\""
Sep 14 10:14:28.003011 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:28.002952    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n"]
Sep 14 10:14:28.013300 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:28.013269    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:14:28.113687 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:28.113644    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:14:28.114176 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:28.114151    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51e15aea-5153-46bc-b34c-e186355d15f6-workload-socket\") pod \"e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n\" (UID: \"51e15aea-5153-46bc-b34c-e186355d15f6\") " pod="openshift-ingress/e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n"
Sep 14 10:14:28.114256 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:28.114194    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51e15aea-5153-46bc-b34c-e186355d15f6-istio-envoy\") pod \"e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n\" (UID: \"51e15aea-5153-46bc-b34c-e186355d15f6\") " pod="openshift-ingress/e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n"
Sep 14 10:14:28.114294 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:28.114250    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jdng\" (UniqueName: \"kubernetes.io/projected/51e15aea-5153-46bc-b34c-e186355d15f6-kube-api-access-4jdng\") pod \"e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n\" (UID: \"51e15aea-5153-46bc-b34c-e186355d15f6\") " pod="openshift-ingress/e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n"
Sep 14 10:14:28.114339 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:28.114326    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51e15aea-5153-46bc-b34c-e186355d15f6-credential-socket\") pod \"e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n\" (UID: \"51e15aea-5153-46bc-b34c-e186355d15f6\") " pod="openshift-ingress/e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n"
Sep 14 10:14:28.114376 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:28.114347    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51e15aea-5153-46bc-b34c-e186355d15f6-istio-podinfo\") pod \"e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n\" (UID: \"51e15aea-5153-46bc-b34c-e186355d15f6\") " pod="openshift-ingress/e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n"
Sep 14 10:14:28.114376 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:28.114364    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51e15aea-5153-46bc-b34c-e186355d15f6-istiod-ca-cert\") pod \"e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n\" (UID: \"51e15aea-5153-46bc-b34c-e186355d15f6\") " pod="openshift-ingress/e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n"
Sep 14 10:14:28.114448 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:28.114382    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51e15aea-5153-46bc-b34c-e186355d15f6-workload-certs\") pod \"e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n\" (UID: \"51e15aea-5153-46bc-b34c-e186355d15f6\") " pod="openshift-ingress/e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n"
Sep 14 10:14:28.114486 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:28.114469    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51e15aea-5153-46bc-b34c-e186355d15f6-istio-token\") pod \"e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n\" (UID: \"51e15aea-5153-46bc-b34c-e186355d15f6\") " pod="openshift-ingress/e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n"
Sep 14 10:14:28.114521 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:28.114500    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51e15aea-5153-46bc-b34c-e186355d15f6-istio-data\") pod \"e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n\" (UID: \"51e15aea-5153-46bc-b34c-e186355d15f6\") " pod="openshift-ingress/e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n"
Sep 14 10:14:28.215480 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:28.215433    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51e15aea-5153-46bc-b34c-e186355d15f6-istio-token\") pod \"e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n\" (UID: \"51e15aea-5153-46bc-b34c-e186355d15f6\") " pod="openshift-ingress/e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n"
Sep 14 10:14:28.215729 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:28.215494    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51e15aea-5153-46bc-b34c-e186355d15f6-istio-data\") pod \"e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n\" (UID: \"51e15aea-5153-46bc-b34c-e186355d15f6\") " pod="openshift-ingress/e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n"
Sep 14 10:14:28.215729 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:28.215539    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51e15aea-5153-46bc-b34c-e186355d15f6-workload-socket\") pod \"e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n\" (UID: \"51e15aea-5153-46bc-b34c-e186355d15f6\") " pod="openshift-ingress/e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n"
Sep 14 10:14:28.215729 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:28.215577    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51e15aea-5153-46bc-b34c-e186355d15f6-istio-envoy\") pod \"e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n\" (UID: \"51e15aea-5153-46bc-b34c-e186355d15f6\") " pod="openshift-ingress/e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n"
Sep 14 10:14:28.215729 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:28.215608    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jdng\" (UniqueName: \"kubernetes.io/projected/51e15aea-5153-46bc-b34c-e186355d15f6-kube-api-access-4jdng\") pod \"e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n\" (UID: \"51e15aea-5153-46bc-b34c-e186355d15f6\") " pod="openshift-ingress/e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n"
Sep 14 10:14:28.215729 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:28.215691    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51e15aea-5153-46bc-b34c-e186355d15f6-credential-socket\") pod \"e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n\" (UID: \"51e15aea-5153-46bc-b34c-e186355d15f6\") " pod="openshift-ingress/e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n"
Sep 14 10:14:28.215729 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:28.215716    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51e15aea-5153-46bc-b34c-e186355d15f6-istio-podinfo\") pod \"e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n\" (UID: \"51e15aea-5153-46bc-b34c-e186355d15f6\") " pod="openshift-ingress/e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n"
Sep 14 10:14:28.216120 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:28.215742    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51e15aea-5153-46bc-b34c-e186355d15f6-istiod-ca-cert\") pod \"e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n\" (UID: \"51e15aea-5153-46bc-b34c-e186355d15f6\") " pod="openshift-ingress/e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n"
Sep 14 10:14:28.216120 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:28.215846    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51e15aea-5153-46bc-b34c-e186355d15f6-workload-certs\") pod \"e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n\" (UID: \"51e15aea-5153-46bc-b34c-e186355d15f6\") " pod="openshift-ingress/e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n"
Sep 14 10:14:28.216120 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:28.215955    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51e15aea-5153-46bc-b34c-e186355d15f6-istio-data\") pod \"e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n\" (UID: \"51e15aea-5153-46bc-b34c-e186355d15f6\") " pod="openshift-ingress/e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n"
Sep 14 10:14:28.216120 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:28.215968    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51e15aea-5153-46bc-b34c-e186355d15f6-workload-socket\") pod \"e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n\" (UID: \"51e15aea-5153-46bc-b34c-e186355d15f6\") " pod="openshift-ingress/e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n"
Sep 14 10:14:28.216337 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:28.216261    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51e15aea-5153-46bc-b34c-e186355d15f6-workload-certs\") pod \"e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n\" (UID: \"51e15aea-5153-46bc-b34c-e186355d15f6\") " pod="openshift-ingress/e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n"
Sep 14 10:14:28.216433 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:28.216411    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51e15aea-5153-46bc-b34c-e186355d15f6-credential-socket\") pod \"e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n\" (UID: \"51e15aea-5153-46bc-b34c-e186355d15f6\") " pod="openshift-ingress/e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n"
Sep 14 10:14:28.216567 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:28.216535    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51e15aea-5153-46bc-b34c-e186355d15f6-istiod-ca-cert\") pod \"e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n\" (UID: \"51e15aea-5153-46bc-b34c-e186355d15f6\") " pod="openshift-ingress/e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n"
Sep 14 10:14:28.218218 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:28.218191    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51e15aea-5153-46bc-b34c-e186355d15f6-istio-envoy\") pod \"e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n\" (UID: \"51e15aea-5153-46bc-b34c-e186355d15f6\") " pod="openshift-ingress/e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n"
Sep 14 10:14:28.218374 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:28.218355    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51e15aea-5153-46bc-b34c-e186355d15f6-istio-podinfo\") pod \"e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n\" (UID: \"51e15aea-5153-46bc-b34c-e186355d15f6\") " pod="openshift-ingress/e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n"
Sep 14 10:14:28.223554 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:28.223512    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51e15aea-5153-46bc-b34c-e186355d15f6-istio-token\") pod \"e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n\" (UID: \"51e15aea-5153-46bc-b34c-e186355d15f6\") " pod="openshift-ingress/e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n"
Sep 14 10:14:28.223730 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:28.223711    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jdng\" (UniqueName: \"kubernetes.io/projected/51e15aea-5153-46bc-b34c-e186355d15f6-kube-api-access-4jdng\") pod \"e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n\" (UID: \"51e15aea-5153-46bc-b34c-e186355d15f6\") " pod="openshift-ingress/e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n"
Sep 14 10:14:28.302478 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:28.302381    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n"
Sep 14 10:14:28.405576 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:28.405532    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:14:28.585466 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:28.585431    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n"]
Sep 14 10:14:28.587183 ip-10-0-132-4 kubenswrapper[2563]: W0914 10:14:28.587154    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51e15aea_5153_46bc_b34c_e186355d15f6.slice/crio-d766e62891caef2abbc564ddfd909554d01b7749c0adc46b425f2d3052e5fb72 WatchSource:0}: Error finding container d766e62891caef2abbc564ddfd909554d01b7749c0adc46b425f2d3052e5fb72: Status 404 returned error can't find the container with id d766e62891caef2abbc564ddfd909554d01b7749c0adc46b425f2d3052e5fb72
Sep 14 10:14:28.589927 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:28.589891    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 10:14:28.590056 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:28.589983    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 10:14:28.590056 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:28.590044    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 10:14:28.765285 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:28.765247    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:14:29.366121 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:29.366081    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n" event={"ID":"51e15aea-5153-46bc-b34c-e186355d15f6","Type":"ContainerStarted","Data":"90c883ed9147b08e0ee074058baffc8bdfdd525132bd387c0c32b1a9933471a8"}
Sep 14 10:14:29.366319 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:29.366142    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n" event={"ID":"51e15aea-5153-46bc-b34c-e186355d15f6","Type":"ContainerStarted","Data":"d766e62891caef2abbc564ddfd909554d01b7749c0adc46b425f2d3052e5fb72"}
Sep 14 10:14:29.422113 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:29.422044    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n" podStartSLOduration=2.422022665 podStartE2EDuration="2.422022665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:14:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:14:29.419294234 +0000 UTC m=+1858.458968854" watchObservedRunningTime="2026-09-14 10:14:29.422022665 +0000 UTC m=+1858.461697277"
Sep 14 10:14:30.303084 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:30.303042    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n"
Sep 14 10:14:30.308619 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:30.308585    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n"
Sep 14 10:14:30.372787 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:30.372748    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n"
Sep 14 10:14:30.373836 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:30.373811    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n"
Sep 14 10:14:30.610725 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:30.610631    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:14:31.314673 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:31.314638    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:14:31.406896 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:31.406864    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:14:31.505843 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:31.505807    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:14:32.299645 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:32.299537    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:14:33.181727 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:33.181693    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:14:34.007890 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:34.007852    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:14:34.107262 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:34.107218    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:14:34.352328 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:34.352234    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:14:34.637819 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:34.637716    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:14:34.734865 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:34.734830    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:14:39.218552 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:39.218513    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:14:39.324783 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:39.324745    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:14:50.502904 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:50.502868    2563 generic.go:356] "Generic (PLEG): container finished" podID="51e15aea-5153-46bc-b34c-e186355d15f6" containerID="90c883ed9147b08e0ee074058baffc8bdfdd525132bd387c0c32b1a9933471a8" exitCode=0
Sep 14 10:14:50.503390 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:50.502950    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n" event={"ID":"51e15aea-5153-46bc-b34c-e186355d15f6","Type":"ContainerDied","Data":"90c883ed9147b08e0ee074058baffc8bdfdd525132bd387c0c32b1a9933471a8"}
Sep 14 10:14:50.503458 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:50.503443    2563 scope.go:117] "RemoveContainer" containerID="90c883ed9147b08e0ee074058baffc8bdfdd525132bd387c0c32b1a9933471a8"
Sep 14 10:14:50.504683 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:50.504652    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 10:14:50.504758 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:50.504730    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 10:14:50.504799 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:50.504765    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 10:14:51.511846 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:51.511811    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n" event={"ID":"51e15aea-5153-46bc-b34c-e186355d15f6","Type":"ContainerStarted","Data":"89454800b3d11bd90e0eaaeaa4a2637f6cd3464efc29df3ea976d2959f7f2e3f"}
Sep 14 10:14:52.303296 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:52.303255    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n"
Sep 14 10:14:52.303570 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:52.303549    2563 patch_prober.go:28] interesting pod/e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Sep 14 10:14:52.303637 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:52.303594    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n" podUID="51e15aea-5153-46bc-b34c-e186355d15f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Sep 14 10:14:53.303607 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:53.303501    2563 patch_prober.go:28] interesting pod/e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Sep 14 10:14:53.303607 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:53.303567    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n" podUID="51e15aea-5153-46bc-b34c-e186355d15f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Sep 14 10:14:54.303275 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:54.303235    2563 patch_prober.go:28] interesting pod/e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Sep 14 10:14:54.303466 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:54.303299    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n" podUID="51e15aea-5153-46bc-b34c-e186355d15f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Sep 14 10:14:55.303323 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:55.303279    2563 patch_prober.go:28] interesting pod/e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Sep 14 10:14:55.303832 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:55.303388    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n" podUID="51e15aea-5153-46bc-b34c-e186355d15f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Sep 14 10:14:56.296429 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:56.296394    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:14:56.303632 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:56.303592    2563 patch_prober.go:28] interesting pod/e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Sep 14 10:14:56.304037 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:56.303661    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n" podUID="51e15aea-5153-46bc-b34c-e186355d15f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Sep 14 10:14:56.400300 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:56.400262    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:14:57.296316 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:57.296280    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:14:57.307706 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:57.307674    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n"
Sep 14 10:14:57.308195 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:57.307971    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n"
Sep 14 10:14:57.308784 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:57.308768    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n"
Sep 14 10:14:58.031148 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:58.031110    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:14:58.891359 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:58.891322    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:14:59.603686 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:14:59.603641    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:15:00.143748 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:00.143695    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-78742b-29823015-dmfb2"]
Sep 14 10:15:00.146940 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:00.146909    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-78742b-29823015-dmfb2"
Sep 14 10:15:00.149351 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:00.149324    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gf4xj\""
Sep 14 10:15:00.150937 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:00.150913    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2c2a39-29823015-5q59k"]
Sep 14 10:15:00.153670 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:00.153642    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2c2a39-29823015-5q59k"
Sep 14 10:15:00.155937 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:00.155908    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f4d054-29823015-64dc9"]
Sep 14 10:15:00.158501 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:00.158471    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-78742b-29823015-dmfb2"
Sep 14 10:15:00.158993 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:00.158966    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f4d054-29823015-64dc9"
Sep 14 10:15:00.160300 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:00.160277    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-78742b-29823015-dmfb2"]
Sep 14 10:15:00.163306 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:00.163274    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29823015-v5lw8"]
Sep 14 10:15:00.166602 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:00.166574    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29823015-v5lw8"
Sep 14 10:15:00.170797 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:00.170772    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2c2a39-29823015-5q59k"
Sep 14 10:15:00.186460 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:00.186418    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2c2a39-29823015-5q59k"]
Sep 14 10:15:00.209156 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:00.209116    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f4d054-29823015-64dc9"]
Sep 14 10:15:00.217705 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:00.217661    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29823015-v5lw8"]
Sep 14 10:15:00.230661 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:00.229676    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f4d054-29823015-64dc9"
Sep 14 10:15:00.238906 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:00.237678    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29823015-v5lw8"
Sep 14 10:15:00.449343 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:00.449315    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-78742b-29823015-dmfb2"]
Sep 14 10:15:00.450966 ip-10-0-132-4 kubenswrapper[2563]: W0914 10:15:00.450896    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24f87fcf_4c99_4330_9c33_89871484ff53.slice/crio-3f5e31301dd3a5d576c6f01bdade1bc8567dac448f3850328a5a096661ce87b2 WatchSource:0}: Error finding container 3f5e31301dd3a5d576c6f01bdade1bc8567dac448f3850328a5a096661ce87b2: Status 404 returned error can't find the container with id 3f5e31301dd3a5d576c6f01bdade1bc8567dac448f3850328a5a096661ce87b2
Sep 14 10:15:00.496962 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:00.496926    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2c2a39-29823015-5q59k"]
Sep 14 10:15:00.498829 ip-10-0-132-4 kubenswrapper[2563]: W0914 10:15:00.498795    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6683dc2_8a3f_40c9_8b86_0045280e96e1.slice/crio-a8650ad223ee96bbcb952fd0c5905671f6c88f6dd93486dfc2051aefb770e3a1 WatchSource:0}: Error finding container a8650ad223ee96bbcb952fd0c5905671f6c88f6dd93486dfc2051aefb770e3a1: Status 404 returned error can't find the container with id a8650ad223ee96bbcb952fd0c5905671f6c88f6dd93486dfc2051aefb770e3a1
Sep 14 10:15:00.542939 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:00.542911    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f4d054-29823015-64dc9"]
Sep 14 10:15:00.545154 ip-10-0-132-4 kubenswrapper[2563]: W0914 10:15:00.545115    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe625209_24e3_4c43_8595_2cf4842a92a0.slice/crio-79b59899f2dfaa50b8fe7b874f14d86a89d40585440032d6ef5c2091dca2ac4d WatchSource:0}: Error finding container 79b59899f2dfaa50b8fe7b874f14d86a89d40585440032d6ef5c2091dca2ac4d: Status 404 returned error can't find the container with id 79b59899f2dfaa50b8fe7b874f14d86a89d40585440032d6ef5c2091dca2ac4d
Sep 14 10:15:00.561210 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:00.561179    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29823015-v5lw8"]
Sep 14 10:15:00.563369 ip-10-0-132-4 kubenswrapper[2563]: W0914 10:15:00.563342    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58c33d07_7ff3_469f_abc0_95328375ef7e.slice/crio-b1414e24de5822877f69ca01416e621dadc71fc63002ec710dedb1634d232724 WatchSource:0}: Error finding container b1414e24de5822877f69ca01416e621dadc71fc63002ec710dedb1634d232724: Status 404 returned error can't find the container with id b1414e24de5822877f69ca01416e621dadc71fc63002ec710dedb1634d232724
Sep 14 10:15:00.580669 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:00.580637    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f4d054-29823015-64dc9" event={"ID":"be625209-24e3-4c43-8595-2cf4842a92a0","Type":"ContainerStarted","Data":"79b59899f2dfaa50b8fe7b874f14d86a89d40585440032d6ef5c2091dca2ac4d"}
Sep 14 10:15:00.582388 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:00.582346    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29823015-v5lw8" event={"ID":"58c33d07-7ff3-469f-abc0-95328375ef7e","Type":"ContainerStarted","Data":"b1414e24de5822877f69ca01416e621dadc71fc63002ec710dedb1634d232724"}
Sep 14 10:15:00.584120 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:00.584082    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-78742b-29823015-dmfb2" event={"ID":"24f87fcf-4c99-4330-9c33-89871484ff53","Type":"ContainerStarted","Data":"3f5e31301dd3a5d576c6f01bdade1bc8567dac448f3850328a5a096661ce87b2"}
Sep 14 10:15:00.585627 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:00.585598    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2c2a39-29823015-5q59k" event={"ID":"e6683dc2-8a3f-40c9-8b86-0045280e96e1","Type":"ContainerStarted","Data":"a8650ad223ee96bbcb952fd0c5905671f6c88f6dd93486dfc2051aefb770e3a1"}
Sep 14 10:15:01.592981 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:01.592943    2563 generic.go:356] "Generic (PLEG): container finished" podID="58c33d07-7ff3-469f-abc0-95328375ef7e" containerID="e688bbc8ec0921466e5ceb94ebf5c1de78bf4b0a24080a8f11e36a9ae495af2f" exitCode=0
Sep 14 10:15:01.593481 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:01.593042    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29823015-v5lw8" event={"ID":"58c33d07-7ff3-469f-abc0-95328375ef7e","Type":"ContainerDied","Data":"e688bbc8ec0921466e5ceb94ebf5c1de78bf4b0a24080a8f11e36a9ae495af2f"}
Sep 14 10:15:01.594911 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:01.594881    2563 generic.go:356] "Generic (PLEG): container finished" podID="24f87fcf-4c99-4330-9c33-89871484ff53" containerID="9fb0bed13a80a6119dcf8ec136f5aa8f249193f8f4164ae0c980eda9b5036632" exitCode=0
Sep 14 10:15:01.595075 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:01.594970    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-78742b-29823015-dmfb2" event={"ID":"24f87fcf-4c99-4330-9c33-89871484ff53","Type":"ContainerDied","Data":"9fb0bed13a80a6119dcf8ec136f5aa8f249193f8f4164ae0c980eda9b5036632"}
Sep 14 10:15:01.596790 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:01.596756    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2c2a39-29823015-5q59k" event={"ID":"e6683dc2-8a3f-40c9-8b86-0045280e96e1","Type":"ContainerStarted","Data":"728ecf78453282da53262fe623d23090745bdf57108e49e288dd06946fc7674e"}
Sep 14 10:15:01.598592 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:01.598535    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f4d054-29823015-64dc9" event={"ID":"be625209-24e3-4c43-8595-2cf4842a92a0","Type":"ContainerStarted","Data":"c15a4c144ca1056199a3d0633999d68d18e829e8516ca02e37c00a73a0e63543"}
Sep 14 10:15:01.636861 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:01.636797    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f4d054-29823015-64dc9" podStartSLOduration=1.636776127 podStartE2EDuration="1.636776127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:15:01.635678119 +0000 UTC m=+1890.675352743" watchObservedRunningTime="2026-09-14 10:15:01.636776127 +0000 UTC m=+1890.676450758"
Sep 14 10:15:01.652791 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:01.652731    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2c2a39-29823015-5q59k" podStartSLOduration=1.6527113839999998 podStartE2EDuration="1.652711384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:15:01.64841735 +0000 UTC m=+1890.688091986" watchObservedRunningTime="2026-09-14 10:15:01.652711384 +0000 UTC m=+1890.692386005"
Sep 14 10:15:02.606433 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:02.606396    2563 generic.go:356] "Generic (PLEG): container finished" podID="e6683dc2-8a3f-40c9-8b86-0045280e96e1" containerID="728ecf78453282da53262fe623d23090745bdf57108e49e288dd06946fc7674e" exitCode=0
Sep 14 10:15:02.606878 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:02.606488    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2c2a39-29823015-5q59k" event={"ID":"e6683dc2-8a3f-40c9-8b86-0045280e96e1","Type":"ContainerDied","Data":"728ecf78453282da53262fe623d23090745bdf57108e49e288dd06946fc7674e"}
Sep 14 10:15:02.608234 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:02.608202    2563 generic.go:356] "Generic (PLEG): container finished" podID="be625209-24e3-4c43-8595-2cf4842a92a0" containerID="c15a4c144ca1056199a3d0633999d68d18e829e8516ca02e37c00a73a0e63543" exitCode=0
Sep 14 10:15:02.608404 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:02.608283    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f4d054-29823015-64dc9" event={"ID":"be625209-24e3-4c43-8595-2cf4842a92a0","Type":"ContainerDied","Data":"c15a4c144ca1056199a3d0633999d68d18e829e8516ca02e37c00a73a0e63543"}
Sep 14 10:15:02.762525 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:02.762493    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29823015-v5lw8"
Sep 14 10:15:02.807971 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:02.807942    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-78742b-29823015-dmfb2"
Sep 14 10:15:03.616354 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:03.616318    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29823015-v5lw8"
Sep 14 10:15:03.616354 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:03.616342    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29823015-v5lw8" event={"ID":"58c33d07-7ff3-469f-abc0-95328375ef7e","Type":"ContainerDied","Data":"b1414e24de5822877f69ca01416e621dadc71fc63002ec710dedb1634d232724"}
Sep 14 10:15:03.616840 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:03.616399    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1414e24de5822877f69ca01416e621dadc71fc63002ec710dedb1634d232724"
Sep 14 10:15:03.618100 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:03.618051    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-78742b-29823015-dmfb2"
Sep 14 10:15:03.618241 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:03.618050    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-78742b-29823015-dmfb2" event={"ID":"24f87fcf-4c99-4330-9c33-89871484ff53","Type":"ContainerDied","Data":"3f5e31301dd3a5d576c6f01bdade1bc8567dac448f3850328a5a096661ce87b2"}
Sep 14 10:15:03.618241 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:03.618184    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f5e31301dd3a5d576c6f01bdade1bc8567dac448f3850328a5a096661ce87b2"
Sep 14 10:15:03.777033 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:03.776975    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f4d054-29823015-64dc9"
Sep 14 10:15:03.780887 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:03.780859    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2c2a39-29823015-5q59k"
Sep 14 10:15:04.629360 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:04.629328    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2c2a39-29823015-5q59k"
Sep 14 10:15:04.629360 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:04.629335    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2c2a39-29823015-5q59k" event={"ID":"e6683dc2-8a3f-40c9-8b86-0045280e96e1","Type":"ContainerDied","Data":"a8650ad223ee96bbcb952fd0c5905671f6c88f6dd93486dfc2051aefb770e3a1"}
Sep 14 10:15:04.629957 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:04.629390    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8650ad223ee96bbcb952fd0c5905671f6c88f6dd93486dfc2051aefb770e3a1"
Sep 14 10:15:04.631107 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:04.631081    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f4d054-29823015-64dc9"
Sep 14 10:15:04.631281 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:04.631112    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f4d054-29823015-64dc9" event={"ID":"be625209-24e3-4c43-8595-2cf4842a92a0","Type":"ContainerDied","Data":"79b59899f2dfaa50b8fe7b874f14d86a89d40585440032d6ef5c2091dca2ac4d"}
Sep 14 10:15:04.631281 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:04.631160    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79b59899f2dfaa50b8fe7b874f14d86a89d40585440032d6ef5c2091dca2ac4d"
Sep 14 10:15:07.854614 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:07.854570    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-78742b/ipp-route-78742b-kserve-779f8dccff-x8bqx"]
Sep 14 10:15:07.855283 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:07.855258    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24f87fcf-4c99-4330-9c33-89871484ff53" containerName="cleanup"
Sep 14 10:15:07.855370 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:07.855287    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="24f87fcf-4c99-4330-9c33-89871484ff53" containerName="cleanup"
Sep 14 10:15:07.855370 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:07.855303    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58c33d07-7ff3-469f-abc0-95328375ef7e" containerName="cleanup"
Sep 14 10:15:07.855370 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:07.855312    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="58c33d07-7ff3-469f-abc0-95328375ef7e" containerName="cleanup"
Sep 14 10:15:07.855370 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:07.855336    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6683dc2-8a3f-40c9-8b86-0045280e96e1" containerName="cleanup"
Sep 14 10:15:07.855370 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:07.855345    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6683dc2-8a3f-40c9-8b86-0045280e96e1" containerName="cleanup"
Sep 14 10:15:07.855370 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:07.855357    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be625209-24e3-4c43-8595-2cf4842a92a0" containerName="cleanup"
Sep 14 10:15:07.855370 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:07.855366    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="be625209-24e3-4c43-8595-2cf4842a92a0" containerName="cleanup"
Sep 14 10:15:07.855745 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:07.855487    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="58c33d07-7ff3-469f-abc0-95328375ef7e" containerName="cleanup"
Sep 14 10:15:07.855745 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:07.855497    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6683dc2-8a3f-40c9-8b86-0045280e96e1" containerName="cleanup"
Sep 14 10:15:07.855745 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:07.855508    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="24f87fcf-4c99-4330-9c33-89871484ff53" containerName="cleanup"
Sep 14 10:15:07.855745 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:07.855517    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="be625209-24e3-4c43-8595-2cf4842a92a0" containerName="cleanup"
Sep 14 10:15:07.859285 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:07.859252    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-78742b/ipp-route-78742b-kserve-779f8dccff-x8bqx"
Sep 14 10:15:07.861606 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:07.861571    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-78742b\"/\"openshift-service-ca.crt\""
Sep 14 10:15:07.862176 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:07.862152    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-78742b\"/\"default-dockercfg-6rhj5\""
Sep 14 10:15:07.862313 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:07.862229    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-78742b\"/\"ipp-route-78742b-kserve-self-signed-certs\""
Sep 14 10:15:07.862313 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:07.862257    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-78742b\"/\"kube-root-ca.crt\""
Sep 14 10:15:07.867483 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:07.867450    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-78742b/ipp-route-78742b-kserve-779f8dccff-x8bqx"]
Sep 14 10:15:07.886847 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:07.886788    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:15:07.918077 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:07.918016    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4663313e-579a-4b8d-9c26-1a10129d8eb9-dshm\") pod \"ipp-route-78742b-kserve-779f8dccff-x8bqx\" (UID: \"4663313e-579a-4b8d-9c26-1a10129d8eb9\") " pod="ai-tenant-e2e-ipp-a-78742b/ipp-route-78742b-kserve-779f8dccff-x8bqx"
Sep 14 10:15:07.918225 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:07.918119    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4663313e-579a-4b8d-9c26-1a10129d8eb9-tls-certs\") pod \"ipp-route-78742b-kserve-779f8dccff-x8bqx\" (UID: \"4663313e-579a-4b8d-9c26-1a10129d8eb9\") " pod="ai-tenant-e2e-ipp-a-78742b/ipp-route-78742b-kserve-779f8dccff-x8bqx"
Sep 14 10:15:07.918225 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:07.918152    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4663313e-579a-4b8d-9c26-1a10129d8eb9-tmp-dir\") pod \"ipp-route-78742b-kserve-779f8dccff-x8bqx\" (UID: \"4663313e-579a-4b8d-9c26-1a10129d8eb9\") " pod="ai-tenant-e2e-ipp-a-78742b/ipp-route-78742b-kserve-779f8dccff-x8bqx"
Sep 14 10:15:07.918298 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:07.918223    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4663313e-579a-4b8d-9c26-1a10129d8eb9-model-cache\") pod \"ipp-route-78742b-kserve-779f8dccff-x8bqx\" (UID: \"4663313e-579a-4b8d-9c26-1a10129d8eb9\") " pod="ai-tenant-e2e-ipp-a-78742b/ipp-route-78742b-kserve-779f8dccff-x8bqx"
Sep 14 10:15:07.918298 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:07.918247    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4663313e-579a-4b8d-9c26-1a10129d8eb9-home\") pod \"ipp-route-78742b-kserve-779f8dccff-x8bqx\" (UID: \"4663313e-579a-4b8d-9c26-1a10129d8eb9\") " pod="ai-tenant-e2e-ipp-a-78742b/ipp-route-78742b-kserve-779f8dccff-x8bqx"
Sep 14 10:15:07.918298 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:07.918286    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdzwf\" (UniqueName: \"kubernetes.io/projected/4663313e-579a-4b8d-9c26-1a10129d8eb9-kube-api-access-vdzwf\") pod \"ipp-route-78742b-kserve-779f8dccff-x8bqx\" (UID: \"4663313e-579a-4b8d-9c26-1a10129d8eb9\") " pod="ai-tenant-e2e-ipp-a-78742b/ipp-route-78742b-kserve-779f8dccff-x8bqx"
Sep 14 10:15:08.019822 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:08.019773    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4663313e-579a-4b8d-9c26-1a10129d8eb9-tls-certs\") pod \"ipp-route-78742b-kserve-779f8dccff-x8bqx\" (UID: \"4663313e-579a-4b8d-9c26-1a10129d8eb9\") " pod="ai-tenant-e2e-ipp-a-78742b/ipp-route-78742b-kserve-779f8dccff-x8bqx"
Sep 14 10:15:08.019822 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:08.019829    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4663313e-579a-4b8d-9c26-1a10129d8eb9-tmp-dir\") pod \"ipp-route-78742b-kserve-779f8dccff-x8bqx\" (UID: \"4663313e-579a-4b8d-9c26-1a10129d8eb9\") " pod="ai-tenant-e2e-ipp-a-78742b/ipp-route-78742b-kserve-779f8dccff-x8bqx"
Sep 14 10:15:08.020126 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:08.019870    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4663313e-579a-4b8d-9c26-1a10129d8eb9-model-cache\") pod \"ipp-route-78742b-kserve-779f8dccff-x8bqx\" (UID: \"4663313e-579a-4b8d-9c26-1a10129d8eb9\") " pod="ai-tenant-e2e-ipp-a-78742b/ipp-route-78742b-kserve-779f8dccff-x8bqx"
Sep 14 10:15:08.020126 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:08.019887    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4663313e-579a-4b8d-9c26-1a10129d8eb9-home\") pod \"ipp-route-78742b-kserve-779f8dccff-x8bqx\" (UID: \"4663313e-579a-4b8d-9c26-1a10129d8eb9\") " pod="ai-tenant-e2e-ipp-a-78742b/ipp-route-78742b-kserve-779f8dccff-x8bqx"
Sep 14 10:15:08.020126 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:08.019907    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdzwf\" (UniqueName: \"kubernetes.io/projected/4663313e-579a-4b8d-9c26-1a10129d8eb9-kube-api-access-vdzwf\") pod \"ipp-route-78742b-kserve-779f8dccff-x8bqx\" (UID: \"4663313e-579a-4b8d-9c26-1a10129d8eb9\") " pod="ai-tenant-e2e-ipp-a-78742b/ipp-route-78742b-kserve-779f8dccff-x8bqx"
Sep 14 10:15:08.020126 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:08.019928    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4663313e-579a-4b8d-9c26-1a10129d8eb9-dshm\") pod \"ipp-route-78742b-kserve-779f8dccff-x8bqx\" (UID: \"4663313e-579a-4b8d-9c26-1a10129d8eb9\") " pod="ai-tenant-e2e-ipp-a-78742b/ipp-route-78742b-kserve-779f8dccff-x8bqx"
Sep 14 10:15:08.020443 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:08.020416    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4663313e-579a-4b8d-9c26-1a10129d8eb9-model-cache\") pod \"ipp-route-78742b-kserve-779f8dccff-x8bqx\" (UID: \"4663313e-579a-4b8d-9c26-1a10129d8eb9\") " pod="ai-tenant-e2e-ipp-a-78742b/ipp-route-78742b-kserve-779f8dccff-x8bqx"
Sep 14 10:15:08.020521 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:08.020449    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4663313e-579a-4b8d-9c26-1a10129d8eb9-home\") pod \"ipp-route-78742b-kserve-779f8dccff-x8bqx\" (UID: \"4663313e-579a-4b8d-9c26-1a10129d8eb9\") " pod="ai-tenant-e2e-ipp-a-78742b/ipp-route-78742b-kserve-779f8dccff-x8bqx"
Sep 14 10:15:08.020521 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:08.020421    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4663313e-579a-4b8d-9c26-1a10129d8eb9-tmp-dir\") pod \"ipp-route-78742b-kserve-779f8dccff-x8bqx\" (UID: \"4663313e-579a-4b8d-9c26-1a10129d8eb9\") " pod="ai-tenant-e2e-ipp-a-78742b/ipp-route-78742b-kserve-779f8dccff-x8bqx"
Sep 14 10:15:08.022261 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:08.022234    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4663313e-579a-4b8d-9c26-1a10129d8eb9-dshm\") pod \"ipp-route-78742b-kserve-779f8dccff-x8bqx\" (UID: \"4663313e-579a-4b8d-9c26-1a10129d8eb9\") " pod="ai-tenant-e2e-ipp-a-78742b/ipp-route-78742b-kserve-779f8dccff-x8bqx"
Sep 14 10:15:08.022663 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:08.022631    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4663313e-579a-4b8d-9c26-1a10129d8eb9-tls-certs\") pod \"ipp-route-78742b-kserve-779f8dccff-x8bqx\" (UID: \"4663313e-579a-4b8d-9c26-1a10129d8eb9\") " pod="ai-tenant-e2e-ipp-a-78742b/ipp-route-78742b-kserve-779f8dccff-x8bqx"
Sep 14 10:15:08.031259 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:08.031222    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdzwf\" (UniqueName: \"kubernetes.io/projected/4663313e-579a-4b8d-9c26-1a10129d8eb9-kube-api-access-vdzwf\") pod \"ipp-route-78742b-kserve-779f8dccff-x8bqx\" (UID: \"4663313e-579a-4b8d-9c26-1a10129d8eb9\") " pod="ai-tenant-e2e-ipp-a-78742b/ipp-route-78742b-kserve-779f8dccff-x8bqx"
Sep 14 10:15:08.171798 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:08.171689    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-78742b/ipp-route-78742b-kserve-779f8dccff-x8bqx"
Sep 14 10:15:08.468183 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:08.468145    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-78742b/ipp-route-78742b-kserve-779f8dccff-x8bqx"]
Sep 14 10:15:08.469259 ip-10-0-132-4 kubenswrapper[2563]: W0914 10:15:08.469227    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4663313e_579a_4b8d_9c26_1a10129d8eb9.slice/crio-c00a3e75b056ec72f1f6542d66c1ce3e3ceec63743b0efcf35f1c7228bb0842f WatchSource:0}: Error finding container c00a3e75b056ec72f1f6542d66c1ce3e3ceec63743b0efcf35f1c7228bb0842f: Status 404 returned error can't find the container with id c00a3e75b056ec72f1f6542d66c1ce3e3ceec63743b0efcf35f1c7228bb0842f
Sep 14 10:15:08.658633 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:08.658597    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-78742b/ipp-route-78742b-kserve-779f8dccff-x8bqx" event={"ID":"4663313e-579a-4b8d-9c26-1a10129d8eb9","Type":"ContainerStarted","Data":"71b30f7c78e9aea3a16b3c3f95cb9da3fa0c41697dcef091d1791d7487ac978f"}
Sep 14 10:15:08.658633 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:08.658640    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-78742b/ipp-route-78742b-kserve-779f8dccff-x8bqx" event={"ID":"4663313e-579a-4b8d-9c26-1a10129d8eb9","Type":"ContainerStarted","Data":"c00a3e75b056ec72f1f6542d66c1ce3e3ceec63743b0efcf35f1c7228bb0842f"}
Sep 14 10:15:08.676139 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:08.676052    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-78742b/ipp-route-78742b-kserve-779f8dccff-x8bqx" podStartSLOduration=1.676031676 podStartE2EDuration="1.676031676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:15:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:15:08.674345373 +0000 UTC m=+1897.714020006" watchObservedRunningTime="2026-09-14 10:15:08.676031676 +0000 UTC m=+1897.715706298"
Sep 14 10:15:17.327873 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:17.327834    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:15:18.172286 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:18.172242    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-78742b/ipp-route-78742b-kserve-779f8dccff-x8bqx"
Sep 14 10:15:18.172483 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:18.172305    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-78742b/ipp-route-78742b-kserve-779f8dccff-x8bqx"
Sep 14 10:15:18.185805 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:18.185764    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-78742b/ipp-route-78742b-kserve-779f8dccff-x8bqx"
Sep 14 10:15:18.735318 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:18.735280    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-78742b/ipp-route-78742b-kserve-779f8dccff-x8bqx"
Sep 14 10:15:26.421525 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:26.421480    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:15:26.484028 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:26.483974    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:15:32.603506 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:32.603463    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:15:32.880831 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:32.880734    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2"]
Sep 14 10:15:32.885679 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:32.885652    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2"
Sep 14 10:15:32.887956 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:32.887929    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-51f250-openshift-default-dockercfg-4q6zt\""
Sep 14 10:15:32.896839 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:32.896806    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2"]
Sep 14 10:15:32.906814 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:32.906778    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:15:32.987740 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:32.987700    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6726601c-3ca3-4150-b479-12eb34b0a0dd-workload-certs\") pod \"e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2\" (UID: \"6726601c-3ca3-4150-b479-12eb34b0a0dd\") " pod="openshift-ingress/e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2"
Sep 14 10:15:32.987965 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:32.987757    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6726601c-3ca3-4150-b479-12eb34b0a0dd-workload-socket\") pod \"e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2\" (UID: \"6726601c-3ca3-4150-b479-12eb34b0a0dd\") " pod="openshift-ingress/e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2"
Sep 14 10:15:32.987965 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:32.987833    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6726601c-3ca3-4150-b479-12eb34b0a0dd-istio-data\") pod \"e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2\" (UID: \"6726601c-3ca3-4150-b479-12eb34b0a0dd\") " pod="openshift-ingress/e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2"
Sep 14 10:15:32.987965 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:32.987876    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6726601c-3ca3-4150-b479-12eb34b0a0dd-credential-socket\") pod \"e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2\" (UID: \"6726601c-3ca3-4150-b479-12eb34b0a0dd\") " pod="openshift-ingress/e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2"
Sep 14 10:15:32.987965 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:32.987909    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrzjd\" (UniqueName: \"kubernetes.io/projected/6726601c-3ca3-4150-b479-12eb34b0a0dd-kube-api-access-qrzjd\") pod \"e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2\" (UID: \"6726601c-3ca3-4150-b479-12eb34b0a0dd\") " pod="openshift-ingress/e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2"
Sep 14 10:15:32.987965 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:32.987954    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6726601c-3ca3-4150-b479-12eb34b0a0dd-istiod-ca-cert\") pod \"e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2\" (UID: \"6726601c-3ca3-4150-b479-12eb34b0a0dd\") " pod="openshift-ingress/e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2"
Sep 14 10:15:32.988281 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:32.987990    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6726601c-3ca3-4150-b479-12eb34b0a0dd-istio-envoy\") pod \"e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2\" (UID: \"6726601c-3ca3-4150-b479-12eb34b0a0dd\") " pod="openshift-ingress/e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2"
Sep 14 10:15:32.988281 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:32.988061    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6726601c-3ca3-4150-b479-12eb34b0a0dd-istio-podinfo\") pod \"e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2\" (UID: \"6726601c-3ca3-4150-b479-12eb34b0a0dd\") " pod="openshift-ingress/e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2"
Sep 14 10:15:32.988281 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:32.988092    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6726601c-3ca3-4150-b479-12eb34b0a0dd-istio-token\") pod \"e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2\" (UID: \"6726601c-3ca3-4150-b479-12eb34b0a0dd\") " pod="openshift-ingress/e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2"
Sep 14 10:15:33.088825 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:33.088782    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6726601c-3ca3-4150-b479-12eb34b0a0dd-istio-data\") pod \"e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2\" (UID: \"6726601c-3ca3-4150-b479-12eb34b0a0dd\") " pod="openshift-ingress/e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2"
Sep 14 10:15:33.088825 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:33.088834    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6726601c-3ca3-4150-b479-12eb34b0a0dd-credential-socket\") pod \"e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2\" (UID: \"6726601c-3ca3-4150-b479-12eb34b0a0dd\") " pod="openshift-ingress/e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2"
Sep 14 10:15:33.089146 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:33.088857    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrzjd\" (UniqueName: \"kubernetes.io/projected/6726601c-3ca3-4150-b479-12eb34b0a0dd-kube-api-access-qrzjd\") pod \"e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2\" (UID: \"6726601c-3ca3-4150-b479-12eb34b0a0dd\") " pod="openshift-ingress/e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2"
Sep 14 10:15:33.089146 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:33.088903    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6726601c-3ca3-4150-b479-12eb34b0a0dd-istiod-ca-cert\") pod \"e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2\" (UID: \"6726601c-3ca3-4150-b479-12eb34b0a0dd\") " pod="openshift-ingress/e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2"
Sep 14 10:15:33.089146 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:33.088931    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6726601c-3ca3-4150-b479-12eb34b0a0dd-istio-envoy\") pod \"e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2\" (UID: \"6726601c-3ca3-4150-b479-12eb34b0a0dd\") " pod="openshift-ingress/e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2"
Sep 14 10:15:33.089146 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:33.089062    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6726601c-3ca3-4150-b479-12eb34b0a0dd-istio-podinfo\") pod \"e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2\" (UID: \"6726601c-3ca3-4150-b479-12eb34b0a0dd\") " pod="openshift-ingress/e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2"
Sep 14 10:15:33.089146 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:33.089104    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6726601c-3ca3-4150-b479-12eb34b0a0dd-istio-token\") pod \"e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2\" (UID: \"6726601c-3ca3-4150-b479-12eb34b0a0dd\") " pod="openshift-ingress/e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2"
Sep 14 10:15:33.089409 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:33.089170    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6726601c-3ca3-4150-b479-12eb34b0a0dd-workload-certs\") pod \"e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2\" (UID: \"6726601c-3ca3-4150-b479-12eb34b0a0dd\") " pod="openshift-ingress/e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2"
Sep 14 10:15:33.089409 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:33.089223    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6726601c-3ca3-4150-b479-12eb34b0a0dd-workload-socket\") pod \"e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2\" (UID: \"6726601c-3ca3-4150-b479-12eb34b0a0dd\") " pod="openshift-ingress/e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2"
Sep 14 10:15:33.089409 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:33.089268    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6726601c-3ca3-4150-b479-12eb34b0a0dd-credential-socket\") pod \"e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2\" (UID: \"6726601c-3ca3-4150-b479-12eb34b0a0dd\") " pod="openshift-ingress/e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2"
Sep 14 10:15:33.089409 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:33.089317    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6726601c-3ca3-4150-b479-12eb34b0a0dd-istio-data\") pod \"e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2\" (UID: \"6726601c-3ca3-4150-b479-12eb34b0a0dd\") " pod="openshift-ingress/e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2"
Sep 14 10:15:33.089657 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:33.089637    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6726601c-3ca3-4150-b479-12eb34b0a0dd-workload-certs\") pod \"e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2\" (UID: \"6726601c-3ca3-4150-b479-12eb34b0a0dd\") " pod="openshift-ingress/e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2"
Sep 14 10:15:33.089704 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:33.089667    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6726601c-3ca3-4150-b479-12eb34b0a0dd-workload-socket\") pod \"e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2\" (UID: \"6726601c-3ca3-4150-b479-12eb34b0a0dd\") " pod="openshift-ingress/e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2"
Sep 14 10:15:33.089742 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:33.089700    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6726601c-3ca3-4150-b479-12eb34b0a0dd-istiod-ca-cert\") pod \"e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2\" (UID: \"6726601c-3ca3-4150-b479-12eb34b0a0dd\") " pod="openshift-ingress/e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2"
Sep 14 10:15:33.091348 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:33.091322    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6726601c-3ca3-4150-b479-12eb34b0a0dd-istio-envoy\") pod \"e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2\" (UID: \"6726601c-3ca3-4150-b479-12eb34b0a0dd\") " pod="openshift-ingress/e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2"
Sep 14 10:15:33.091883 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:33.091855    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6726601c-3ca3-4150-b479-12eb34b0a0dd-istio-podinfo\") pod \"e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2\" (UID: \"6726601c-3ca3-4150-b479-12eb34b0a0dd\") " pod="openshift-ingress/e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2"
Sep 14 10:15:33.097294 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:33.097266    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6726601c-3ca3-4150-b479-12eb34b0a0dd-istio-token\") pod \"e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2\" (UID: \"6726601c-3ca3-4150-b479-12eb34b0a0dd\") " pod="openshift-ingress/e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2"
Sep 14 10:15:33.097545 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:33.097520    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrzjd\" (UniqueName: \"kubernetes.io/projected/6726601c-3ca3-4150-b479-12eb34b0a0dd-kube-api-access-qrzjd\") pod \"e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2\" (UID: \"6726601c-3ca3-4150-b479-12eb34b0a0dd\") " pod="openshift-ingress/e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2"
Sep 14 10:15:33.198954 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:33.198848    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2"
Sep 14 10:15:33.325509 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:33.325464    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:15:33.487880 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:33.487841    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2"]
Sep 14 10:15:33.488669 ip-10-0-132-4 kubenswrapper[2563]: W0914 10:15:33.488637    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6726601c_3ca3_4150_b479_12eb34b0a0dd.slice/crio-f96cd19767beb0e68c0d2d13c8595f0bdc596ab962c5fc545309c662c2df6320 WatchSource:0}: Error finding container f96cd19767beb0e68c0d2d13c8595f0bdc596ab962c5fc545309c662c2df6320: Status 404 returned error can't find the container with id f96cd19767beb0e68c0d2d13c8595f0bdc596ab962c5fc545309c662c2df6320
Sep 14 10:15:33.491510 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:33.491466    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 10:15:33.491661 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:33.491566    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 10:15:33.491661 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:33.491611    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 10:15:33.831191 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:33.831077    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2" event={"ID":"6726601c-3ca3-4150-b479-12eb34b0a0dd","Type":"ContainerStarted","Data":"73e1fcb4fdd6cfb9b0edbecee21d5ffe2810398a40778c8caa1fab12f4dfca46"}
Sep 14 10:15:33.831191 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:33.831175    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2" event={"ID":"6726601c-3ca3-4150-b479-12eb34b0a0dd","Type":"ContainerStarted","Data":"f96cd19767beb0e68c0d2d13c8595f0bdc596ab962c5fc545309c662c2df6320"}
Sep 14 10:15:33.852328 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:33.852183    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2" podStartSLOduration=1.8521596169999999 podStartE2EDuration="1.852159617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:15:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:15:33.851642834 +0000 UTC m=+1922.891317457" watchObservedRunningTime="2026-09-14 10:15:33.852159617 +0000 UTC m=+1922.891834238"
Sep 14 10:15:34.199229 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:34.199127    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2"
Sep 14 10:15:34.205330 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:34.205293    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2"
Sep 14 10:15:34.837438 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:34.837406    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2"
Sep 14 10:15:34.838471 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:34.838448    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2"
Sep 14 10:15:35.092882 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:35.092789    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:15:35.769334 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:35.769294    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:15:35.864409 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:35.864371    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:15:35.964284 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:35.964248    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:15:37.113034 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:37.112975    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:15:38.010965 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:38.010928    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:15:38.878726 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:38.878687    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:15:39.620345 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:39.620306    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:15:39.722294 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:39.722256    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:15:40.098021 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:40.097961    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:15:40.194092 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:40.194054    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:15:51.042453 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:51.042404    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:15:55.986705 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:55.986669    2563 generic.go:356] "Generic (PLEG): container finished" podID="6726601c-3ca3-4150-b479-12eb34b0a0dd" containerID="73e1fcb4fdd6cfb9b0edbecee21d5ffe2810398a40778c8caa1fab12f4dfca46" exitCode=0
Sep 14 10:15:55.987173 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:55.986742    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2" event={"ID":"6726601c-3ca3-4150-b479-12eb34b0a0dd","Type":"ContainerDied","Data":"73e1fcb4fdd6cfb9b0edbecee21d5ffe2810398a40778c8caa1fab12f4dfca46"}
Sep 14 10:15:55.987234 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:55.987216    2563 scope.go:117] "RemoveContainer" containerID="73e1fcb4fdd6cfb9b0edbecee21d5ffe2810398a40778c8caa1fab12f4dfca46"
Sep 14 10:15:55.988580 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:55.988547    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 10:15:55.988659 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:55.988622    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 10:15:55.988696 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:55.988673    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 14 10:15:56.995941 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:56.995899    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2" event={"ID":"6726601c-3ca3-4150-b479-12eb34b0a0dd","Type":"ContainerStarted","Data":"6d20dc3aeae38b1986745d4b5b162c386a191476dac48926a0c879a021c105ed"}
Sep 14 10:15:57.199537 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:57.199491    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2"
Sep 14 10:15:57.199906 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:57.199883    2563 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Sep 14 10:15:57.199975 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:57.199953    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2" podUID="6726601c-3ca3-4150-b479-12eb34b0a0dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Sep 14 10:15:58.199457 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:58.199415    2563 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Sep 14 10:15:58.199900 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:58.199483    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2" podUID="6726601c-3ca3-4150-b479-12eb34b0a0dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Sep 14 10:15:59.200194 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:59.200152    2563 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Sep 14 10:15:59.200630 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:15:59.200225    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2" podUID="6726601c-3ca3-4150-b479-12eb34b0a0dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Sep 14 10:16:00.199659 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:00.199618    2563 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Sep 14 10:16:00.199850 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:00.199712    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2" podUID="6726601c-3ca3-4150-b479-12eb34b0a0dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Sep 14 10:16:00.602646 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:00.602607    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:16:01.199567 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:01.199528    2563 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Sep 14 10:16:01.199769 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:01.199584    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2" podUID="6726601c-3ca3-4150-b479-12eb34b0a0dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Sep 14 10:16:02.081414 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:02.081375    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:16:02.200159 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:02.200117    2563 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Sep 14 10:16:02.200357 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:02.200178    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2" podUID="6726601c-3ca3-4150-b479-12eb34b0a0dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Sep 14 10:16:02.892154 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:02.892122    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:16:03.204293 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:03.204207    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2"
Sep 14 10:16:03.204671 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:03.204521    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2"
Sep 14 10:16:03.205386 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:03.205363    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2"
Sep 14 10:16:03.608767 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:03.608727    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:16:04.304510 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:04.304477    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:16:05.200164 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:05.200119    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:16:05.813501 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:05.813469    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:16:05.915051 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:05.914988    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:16:06.471373 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:06.471337    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-51f250-rvmc6"]
Sep 14 10:16:06.473886 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:06.473862    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-51f250-rvmc6"
Sep 14 10:16:06.475873 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:06.475839    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gf4xj\""
Sep 14 10:16:06.476079 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:06.476051    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 14 10:16:06.483267 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:06.483233    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-51f250-rvmc6"]
Sep 14 10:16:06.533251 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:06.533209    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0516a9a2-2c06-40e1-97e8-7b3c3542053e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-51f250-rvmc6\" (UID: \"0516a9a2-2c06-40e1-97e8-7b3c3542053e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-51f250-rvmc6"
Sep 14 10:16:06.634461 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:06.634421    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0516a9a2-2c06-40e1-97e8-7b3c3542053e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-51f250-rvmc6\" (UID: \"0516a9a2-2c06-40e1-97e8-7b3c3542053e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-51f250-rvmc6"
Sep 14 10:16:06.635126 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:06.635102    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0516a9a2-2c06-40e1-97e8-7b3c3542053e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-51f250-rvmc6\" (UID: \"0516a9a2-2c06-40e1-97e8-7b3c3542053e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-51f250-rvmc6"
Sep 14 10:16:06.787937 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:06.787889    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-51f250-rvmc6"
Sep 14 10:16:07.064173 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:07.064136    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-51f250-rvmc6"]
Sep 14 10:16:07.065532 ip-10-0-132-4 kubenswrapper[2563]: W0914 10:16:07.065503    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0516a9a2_2c06_40e1_97e8_7b3c3542053e.slice/crio-b4ef1bb175719af786ab789528151ce5f482da1f4c371262af459ba8ca043a10 WatchSource:0}: Error finding container b4ef1bb175719af786ab789528151ce5f482da1f4c371262af459ba8ca043a10: Status 404 returned error can't find the container with id b4ef1bb175719af786ab789528151ce5f482da1f4c371262af459ba8ca043a10
Sep 14 10:16:07.380639 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:07.380599    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:16:08.077184 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:08.077145    2563 generic.go:356] "Generic (PLEG): container finished" podID="0516a9a2-2c06-40e1-97e8-7b3c3542053e" containerID="262a09290e83d6483616f9137ec61e2f457a0c2f1f4bc1554a6103ead3a2558d" exitCode=0
Sep 14 10:16:08.077606 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:08.077242    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-51f250-rvmc6" event={"ID":"0516a9a2-2c06-40e1-97e8-7b3c3542053e","Type":"ContainerDied","Data":"262a09290e83d6483616f9137ec61e2f457a0c2f1f4bc1554a6103ead3a2558d"}
Sep 14 10:16:08.077606 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:08.077323    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-51f250-rvmc6" event={"ID":"0516a9a2-2c06-40e1-97e8-7b3c3542053e","Type":"ContainerStarted","Data":"b4ef1bb175719af786ab789528151ce5f482da1f4c371262af459ba8ca043a10"}
Sep 14 10:16:08.504663 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:08.504623    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:16:09.229089 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:09.229059    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-51f250-rvmc6"
Sep 14 10:16:09.362566 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:09.362463    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0516a9a2-2c06-40e1-97e8-7b3c3542053e-maas-api-service-ca\") pod \"0516a9a2-2c06-40e1-97e8-7b3c3542053e\" (UID: \"0516a9a2-2c06-40e1-97e8-7b3c3542053e\") "
Sep 14 10:16:09.362853 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:09.362827    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0516a9a2-2c06-40e1-97e8-7b3c3542053e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0516a9a2-2c06-40e1-97e8-7b3c3542053e" (UID: "0516a9a2-2c06-40e1-97e8-7b3c3542053e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:16:09.464438 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:09.464389    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0516a9a2-2c06-40e1-97e8-7b3c3542053e-maas-api-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:16:09.747093 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:09.747056    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:16:10.092303 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:10.092218    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-51f250-rvmc6"
Sep 14 10:16:10.092303 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:10.092256    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-51f250-rvmc6" event={"ID":"0516a9a2-2c06-40e1-97e8-7b3c3542053e","Type":"ContainerDied","Data":"b4ef1bb175719af786ab789528151ce5f482da1f4c371262af459ba8ca043a10"}
Sep 14 10:16:10.092493 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:10.092309    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4ef1bb175719af786ab789528151ce5f482da1f4c371262af459ba8ca043a10"
Sep 14 10:16:11.395782 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:11.395746    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:16:12.651337 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:12.651299    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:16:13.910442 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:13.910405    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:16:17.876387 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:17.876349    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:16:17.958192 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:17.958143    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:16:18.055349 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:18.055317    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:16:18.160870 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:18.160785    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:16:18.161201 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:18.161172    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-51f250-rvmc6"]
Sep 14 10:16:18.172982 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:18.172936    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-51f250-rvmc6"]
Sep 14 10:16:18.279908 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:18.279870    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:16:18.557452 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:18.557415    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:16:18.658924 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:18.658888    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:16:19.482246 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:19.482208    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0516a9a2-2c06-40e1-97e8-7b3c3542053e" path="/var/lib/kubelet/pods/0516a9a2-2c06-40e1-97e8-7b3c3542053e/volumes"
Sep 14 10:16:19.646256 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:19.646216    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:16:20.924087 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:20.924038    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:16:21.794253 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:21.794209    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:16:22.787940 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:22.787910    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:16:25.321319 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:25.321283    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:16:25.492881 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:25.492848    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:16:25.666281 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:25.666188    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2"]
Sep 14 10:16:25.666453 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:25.666423    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2" podUID="6726601c-3ca3-4150-b479-12eb34b0a0dd" containerName="istio-proxy" containerID="cri-o://6d20dc3aeae38b1986745d4b5b162c386a191476dac48926a0c879a021c105ed" gracePeriod=30
Sep 14 10:16:25.782112 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:25.782067    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:16:29.520843 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:29.520801    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78742b-5gpcp"]
Sep 14 10:16:29.521405 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:29.521219    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0516a9a2-2c06-40e1-97e8-7b3c3542053e" containerName="revoke-keys"
Sep 14 10:16:29.521405 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:29.521232    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0516a9a2-2c06-40e1-97e8-7b3c3542053e" containerName="revoke-keys"
Sep 14 10:16:29.521405 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:29.521314    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0516a9a2-2c06-40e1-97e8-7b3c3542053e" containerName="revoke-keys"
Sep 14 10:16:29.523669 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:29.523648    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78742b-5gpcp"
Sep 14 10:16:29.532376 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:29.532352    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gf4xj\""
Sep 14 10:16:29.532498 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:29.532403    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 14 10:16:29.544890 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:29.544849    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78742b-5gpcp"]
Sep 14 10:16:29.554948 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:29.554897    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c8b7935-ed70-456d-bc61-cc4a4b0ba269-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-78742b-5gpcp\" (UID: \"4c8b7935-ed70-456d-bc61-cc4a4b0ba269\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78742b-5gpcp"
Sep 14 10:16:29.656242 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:29.656193    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c8b7935-ed70-456d-bc61-cc4a4b0ba269-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-78742b-5gpcp\" (UID: \"4c8b7935-ed70-456d-bc61-cc4a4b0ba269\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78742b-5gpcp"
Sep 14 10:16:29.656884 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:29.656864    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c8b7935-ed70-456d-bc61-cc4a4b0ba269-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-78742b-5gpcp\" (UID: \"4c8b7935-ed70-456d-bc61-cc4a4b0ba269\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78742b-5gpcp"
Sep 14 10:16:29.752232 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:29.752171    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:16:29.835187 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:29.835094    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78742b-5gpcp"
Sep 14 10:16:30.122163 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:30.122133    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78742b-5gpcp"]
Sep 14 10:16:30.125215 ip-10-0-132-4 kubenswrapper[2563]: W0914 10:16:30.125184    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c8b7935_ed70_456d_bc61_cc4a4b0ba269.slice/crio-82707f7e1447cc2322b243832460dc84d72e232ca3f26e8d844d838a550e37eb WatchSource:0}: Error finding container 82707f7e1447cc2322b243832460dc84d72e232ca3f26e8d844d838a550e37eb: Status 404 returned error can't find the container with id 82707f7e1447cc2322b243832460dc84d72e232ca3f26e8d844d838a550e37eb
Sep 14 10:16:30.223518 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:30.223479    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78742b-5gpcp" event={"ID":"4c8b7935-ed70-456d-bc61-cc4a4b0ba269","Type":"ContainerStarted","Data":"82707f7e1447cc2322b243832460dc84d72e232ca3f26e8d844d838a550e37eb"}
Sep 14 10:16:30.927280 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:30.927200    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2"
Sep 14 10:16:30.969773 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:30.969725    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6726601c-3ca3-4150-b479-12eb34b0a0dd-credential-socket\") pod \"6726601c-3ca3-4150-b479-12eb34b0a0dd\" (UID: \"6726601c-3ca3-4150-b479-12eb34b0a0dd\") "
Sep 14 10:16:30.969961 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:30.969791    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6726601c-3ca3-4150-b479-12eb34b0a0dd-workload-socket\") pod \"6726601c-3ca3-4150-b479-12eb34b0a0dd\" (UID: \"6726601c-3ca3-4150-b479-12eb34b0a0dd\") "
Sep 14 10:16:30.969961 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:30.969825    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6726601c-3ca3-4150-b479-12eb34b0a0dd-workload-certs\") pod \"6726601c-3ca3-4150-b479-12eb34b0a0dd\" (UID: \"6726601c-3ca3-4150-b479-12eb34b0a0dd\") "
Sep 14 10:16:30.969961 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:30.969853    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6726601c-3ca3-4150-b479-12eb34b0a0dd-istio-token\") pod \"6726601c-3ca3-4150-b479-12eb34b0a0dd\" (UID: \"6726601c-3ca3-4150-b479-12eb34b0a0dd\") "
Sep 14 10:16:30.969961 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:30.969886    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6726601c-3ca3-4150-b479-12eb34b0a0dd-istio-envoy\") pod \"6726601c-3ca3-4150-b479-12eb34b0a0dd\" (UID: \"6726601c-3ca3-4150-b479-12eb34b0a0dd\") "
Sep 14 10:16:30.969961 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:30.969941    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6726601c-3ca3-4150-b479-12eb34b0a0dd-istio-data\") pod \"6726601c-3ca3-4150-b479-12eb34b0a0dd\" (UID: \"6726601c-3ca3-4150-b479-12eb34b0a0dd\") "
Sep 14 10:16:30.970290 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:30.969974    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6726601c-3ca3-4150-b479-12eb34b0a0dd-istiod-ca-cert\") pod \"6726601c-3ca3-4150-b479-12eb34b0a0dd\" (UID: \"6726601c-3ca3-4150-b479-12eb34b0a0dd\") "
Sep 14 10:16:30.970290 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:30.970056    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qrzjd\" (UniqueName: \"kubernetes.io/projected/6726601c-3ca3-4150-b479-12eb34b0a0dd-kube-api-access-qrzjd\") pod \"6726601c-3ca3-4150-b479-12eb34b0a0dd\" (UID: \"6726601c-3ca3-4150-b479-12eb34b0a0dd\") "
Sep 14 10:16:30.970290 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:30.970087    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6726601c-3ca3-4150-b479-12eb34b0a0dd-istio-podinfo\") pod \"6726601c-3ca3-4150-b479-12eb34b0a0dd\" (UID: \"6726601c-3ca3-4150-b479-12eb34b0a0dd\") "
Sep 14 10:16:30.970290 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:30.970080    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6726601c-3ca3-4150-b479-12eb34b0a0dd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6726601c-3ca3-4150-b479-12eb34b0a0dd" (UID: "6726601c-3ca3-4150-b479-12eb34b0a0dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:16:30.970290 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:30.970123    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6726601c-3ca3-4150-b479-12eb34b0a0dd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6726601c-3ca3-4150-b479-12eb34b0a0dd" (UID: "6726601c-3ca3-4150-b479-12eb34b0a0dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:16:30.970290 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:30.970188    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6726601c-3ca3-4150-b479-12eb34b0a0dd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6726601c-3ca3-4150-b479-12eb34b0a0dd" (UID: "6726601c-3ca3-4150-b479-12eb34b0a0dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:16:30.970596 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:30.970397    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6726601c-3ca3-4150-b479-12eb34b0a0dd-istio-data" (OuterVolumeSpecName: "istio-data") pod "6726601c-3ca3-4150-b479-12eb34b0a0dd" (UID: "6726601c-3ca3-4150-b479-12eb34b0a0dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:16:30.970596 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:30.970575    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6726601c-3ca3-4150-b479-12eb34b0a0dd-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:16:30.970698 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:30.970596    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6726601c-3ca3-4150-b479-12eb34b0a0dd-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:16:30.970698 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:30.970611    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6726601c-3ca3-4150-b479-12eb34b0a0dd-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:16:30.970698 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:30.970616    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6726601c-3ca3-4150-b479-12eb34b0a0dd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6726601c-3ca3-4150-b479-12eb34b0a0dd" (UID: "6726601c-3ca3-4150-b479-12eb34b0a0dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:16:30.970698 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:30.970625    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6726601c-3ca3-4150-b479-12eb34b0a0dd-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:16:30.972461 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:30.972422    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6726601c-3ca3-4150-b479-12eb34b0a0dd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6726601c-3ca3-4150-b479-12eb34b0a0dd" (UID: "6726601c-3ca3-4150-b479-12eb34b0a0dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:16:30.972461 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:30.972427    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6726601c-3ca3-4150-b479-12eb34b0a0dd-istio-token" (OuterVolumeSpecName: "istio-token") pod "6726601c-3ca3-4150-b479-12eb34b0a0dd" (UID: "6726601c-3ca3-4150-b479-12eb34b0a0dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:16:30.972814 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:30.972793    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6726601c-3ca3-4150-b479-12eb34b0a0dd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6726601c-3ca3-4150-b479-12eb34b0a0dd" (UID: "6726601c-3ca3-4150-b479-12eb34b0a0dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 10:16:30.972857 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:30.972836    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6726601c-3ca3-4150-b479-12eb34b0a0dd-kube-api-access-qrzjd" (OuterVolumeSpecName: "kube-api-access-qrzjd") pod "6726601c-3ca3-4150-b479-12eb34b0a0dd" (UID: "6726601c-3ca3-4150-b479-12eb34b0a0dd"). InnerVolumeSpecName "kube-api-access-qrzjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:16:31.071531 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:31.071487    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6726601c-3ca3-4150-b479-12eb34b0a0dd-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:16:31.071531 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:31.071523    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6726601c-3ca3-4150-b479-12eb34b0a0dd-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:16:31.071531 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:31.071532    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6726601c-3ca3-4150-b479-12eb34b0a0dd-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:16:31.071531 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:31.071541    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrzjd\" (UniqueName: \"kubernetes.io/projected/6726601c-3ca3-4150-b479-12eb34b0a0dd-kube-api-access-qrzjd\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:16:31.071837 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:31.071552    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6726601c-3ca3-4150-b479-12eb34b0a0dd-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:16:31.083523 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:31.083485    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:16:31.231562 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:31.231463    2563 generic.go:356] "Generic (PLEG): container finished" podID="4c8b7935-ed70-456d-bc61-cc4a4b0ba269" containerID="5eb5696cd408662fdc6cd7d3a4c6fcd34f2471009c42a85b689086a48a8d3d59" exitCode=0
Sep 14 10:16:31.231727 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:31.231556    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78742b-5gpcp" event={"ID":"4c8b7935-ed70-456d-bc61-cc4a4b0ba269","Type":"ContainerDied","Data":"5eb5696cd408662fdc6cd7d3a4c6fcd34f2471009c42a85b689086a48a8d3d59"}
Sep 14 10:16:31.233536 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:31.233503    2563 generic.go:356] "Generic (PLEG): container finished" podID="6726601c-3ca3-4150-b479-12eb34b0a0dd" containerID="6d20dc3aeae38b1986745d4b5b162c386a191476dac48926a0c879a021c105ed" exitCode=0
Sep 14 10:16:31.233693 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:31.233552    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2"
Sep 14 10:16:31.233693 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:31.233594    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2" event={"ID":"6726601c-3ca3-4150-b479-12eb34b0a0dd","Type":"ContainerDied","Data":"6d20dc3aeae38b1986745d4b5b162c386a191476dac48926a0c879a021c105ed"}
Sep 14 10:16:31.233693 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:31.233656    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2" event={"ID":"6726601c-3ca3-4150-b479-12eb34b0a0dd","Type":"ContainerDied","Data":"f96cd19767beb0e68c0d2d13c8595f0bdc596ab962c5fc545309c662c2df6320"}
Sep 14 10:16:31.233693 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:31.233684    2563 scope.go:117] "RemoveContainer" containerID="6d20dc3aeae38b1986745d4b5b162c386a191476dac48926a0c879a021c105ed"
Sep 14 10:16:31.244770 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:31.244747    2563 scope.go:117] "RemoveContainer" containerID="73e1fcb4fdd6cfb9b0edbecee21d5ffe2810398a40778c8caa1fab12f4dfca46"
Sep 14 10:16:31.255180 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:31.255154    2563 scope.go:117] "RemoveContainer" containerID="6d20dc3aeae38b1986745d4b5b162c386a191476dac48926a0c879a021c105ed"
Sep 14 10:16:31.255543 ip-10-0-132-4 kubenswrapper[2563]: E0914 10:16:31.255516    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d20dc3aeae38b1986745d4b5b162c386a191476dac48926a0c879a021c105ed\": container with ID starting with 6d20dc3aeae38b1986745d4b5b162c386a191476dac48926a0c879a021c105ed not found: ID does not exist" containerID="6d20dc3aeae38b1986745d4b5b162c386a191476dac48926a0c879a021c105ed"
Sep 14 10:16:31.255646 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:31.255555    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d20dc3aeae38b1986745d4b5b162c386a191476dac48926a0c879a021c105ed"} err="failed to get container status \"6d20dc3aeae38b1986745d4b5b162c386a191476dac48926a0c879a021c105ed\": rpc error: code = NotFound desc = could not find container \"6d20dc3aeae38b1986745d4b5b162c386a191476dac48926a0c879a021c105ed\": container with ID starting with 6d20dc3aeae38b1986745d4b5b162c386a191476dac48926a0c879a021c105ed not found: ID does not exist"
Sep 14 10:16:31.255646 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:31.255582    2563 scope.go:117] "RemoveContainer" containerID="73e1fcb4fdd6cfb9b0edbecee21d5ffe2810398a40778c8caa1fab12f4dfca46"
Sep 14 10:16:31.255927 ip-10-0-132-4 kubenswrapper[2563]: E0914 10:16:31.255908    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73e1fcb4fdd6cfb9b0edbecee21d5ffe2810398a40778c8caa1fab12f4dfca46\": container with ID starting with 73e1fcb4fdd6cfb9b0edbecee21d5ffe2810398a40778c8caa1fab12f4dfca46 not found: ID does not exist" containerID="73e1fcb4fdd6cfb9b0edbecee21d5ffe2810398a40778c8caa1fab12f4dfca46"
Sep 14 10:16:31.255973 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:31.255934    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73e1fcb4fdd6cfb9b0edbecee21d5ffe2810398a40778c8caa1fab12f4dfca46"} err="failed to get container status \"73e1fcb4fdd6cfb9b0edbecee21d5ffe2810398a40778c8caa1fab12f4dfca46\": rpc error: code = NotFound desc = could not find container \"73e1fcb4fdd6cfb9b0edbecee21d5ffe2810398a40778c8caa1fab12f4dfca46\": container with ID starting with 73e1fcb4fdd6cfb9b0edbecee21d5ffe2810398a40778c8caa1fab12f4dfca46 not found: ID does not exist"
Sep 14 10:16:31.280607 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:31.280563    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2"]
Sep 14 10:16:31.295322 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:31.295283    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-51f250-openshift-default-7fc69565f8-bk4b2"]
Sep 14 10:16:31.483257 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:31.483173    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6726601c-3ca3-4150-b479-12eb34b0a0dd" path="/var/lib/kubelet/pods/6726601c-3ca3-4150-b479-12eb34b0a0dd/volumes"
Sep 14 10:16:32.071085 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:32.071055    2563 scope.go:117] "RemoveContainer" containerID="ef2510bd99ebfa87d7766b0a5cbe5a5916d5e78bc001f4db8ea3ab7efd5f2153"
Sep 14 10:16:32.083310 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:32.083287    2563 scope.go:117] "RemoveContainer" containerID="e6224d6bc0c9b22d9c9084532dbff9dfa10a23ef8dcc03f8e7e1d2d706a10474"
Sep 14 10:16:32.093056 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:32.093028    2563 scope.go:117] "RemoveContainer" containerID="345a49e0a3814fdf2eab38ac04b53e4655f39cfe1bbd21a72452635c7815bf45"
Sep 14 10:16:32.103031 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:32.102986    2563 scope.go:117] "RemoveContainer" containerID="9c510680245c45740bf170ee13ecf586dc1580fc533836313fc0d1c440c644c6"
Sep 14 10:16:32.356636 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:32.356559    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:16:32.387113 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:32.387082    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78742b-5gpcp"
Sep 14 10:16:32.486535 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:32.486501    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c8b7935-ed70-456d-bc61-cc4a4b0ba269-maas-api-service-ca\") pod \"4c8b7935-ed70-456d-bc61-cc4a4b0ba269\" (UID: \"4c8b7935-ed70-456d-bc61-cc4a4b0ba269\") "
Sep 14 10:16:32.486881 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:32.486858    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c8b7935-ed70-456d-bc61-cc4a4b0ba269-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4c8b7935-ed70-456d-bc61-cc4a4b0ba269" (UID: "4c8b7935-ed70-456d-bc61-cc4a4b0ba269"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:16:32.587859 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:32.587823    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c8b7935-ed70-456d-bc61-cc4a4b0ba269-maas-api-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:16:33.248552 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:33.248523    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78742b-5gpcp"
Sep 14 10:16:33.248942 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:33.248566    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78742b-5gpcp" event={"ID":"4c8b7935-ed70-456d-bc61-cc4a4b0ba269","Type":"ContainerDied","Data":"82707f7e1447cc2322b243832460dc84d72e232ca3f26e8d844d838a550e37eb"}
Sep 14 10:16:33.248942 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:33.248624    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82707f7e1447cc2322b243832460dc84d72e232ca3f26e8d844d838a550e37eb"
Sep 14 10:16:33.823431 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:33.823389    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:16:35.024509 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:35.024466    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:16:36.724439 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:36.724390    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:16:45.502166 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:45.502122    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:16:45.515088 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:45.515039    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-78742b-29823015-dmfb2"]
Sep 14 10:16:45.522467 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:45.522428    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-78742b-29823015-dmfb2"]
Sep 14 10:16:45.632927 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:45.632897    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:16:45.731872 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:45.731830    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:16:45.822484 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:45.822389    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78742b-5gpcp"]
Sep 14 10:16:45.826888 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:45.826855    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78742b-5gpcp"]
Sep 14 10:16:45.835397 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:45.835367    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:16:45.917784 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:45.917742    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:16:46.234209 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:46.234155    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:16:46.331792 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:46.331743    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:16:47.202818 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:47.202779    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:16:47.482706 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:47.482616    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24f87fcf-4c99-4330-9c33-89871484ff53" path="/var/lib/kubelet/pods/24f87fcf-4c99-4330-9c33-89871484ff53/volumes"
Sep 14 10:16:47.482967 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:47.482953    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c8b7935-ed70-456d-bc61-cc4a4b0ba269" path="/var/lib/kubelet/pods/4c8b7935-ed70-456d-bc61-cc4a4b0ba269/volumes"
Sep 14 10:16:48.436840 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:48.436804    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:16:49.529707 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:49.529663    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:16:51.188175 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:51.188125    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-78742b/ipp-route-78742b-kserve-779f8dccff-x8bqx"]
Sep 14 10:16:51.280938 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:51.280897    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:16:52.593087 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:52.593050    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:16:52.693348 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:52.693317    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:16:53.135947 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:53.135911    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:16:56.729668 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:16:56.729631    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:17:06.208458 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:06.208403    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-78742b/ipp-route-78742b-kserve-779f8dccff-x8bqx" podUID="4663313e-579a-4b8d-9c26-1a10129d8eb9" containerName="main" containerID="cri-o://71b30f7c78e9aea3a16b3c3f95cb9da3fa0c41697dcef091d1791d7487ac978f" gracePeriod=45
Sep 14 10:17:06.470668 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:06.470591    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-78742b/ipp-route-78742b-kserve-779f8dccff-x8bqx"
Sep 14 10:17:06.471676 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:06.471651    2563 generic.go:356] "Generic (PLEG): container finished" podID="4663313e-579a-4b8d-9c26-1a10129d8eb9" containerID="71b30f7c78e9aea3a16b3c3f95cb9da3fa0c41697dcef091d1791d7487ac978f" exitCode=0
Sep 14 10:17:06.471810 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:06.471720    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-78742b/ipp-route-78742b-kserve-779f8dccff-x8bqx" event={"ID":"4663313e-579a-4b8d-9c26-1a10129d8eb9","Type":"ContainerDied","Data":"71b30f7c78e9aea3a16b3c3f95cb9da3fa0c41697dcef091d1791d7487ac978f"}
Sep 14 10:17:06.471810 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:06.471765    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-78742b/ipp-route-78742b-kserve-779f8dccff-x8bqx" event={"ID":"4663313e-579a-4b8d-9c26-1a10129d8eb9","Type":"ContainerDied","Data":"c00a3e75b056ec72f1f6542d66c1ce3e3ceec63743b0efcf35f1c7228bb0842f"}
Sep 14 10:17:06.471810 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:06.471782    2563 scope.go:117] "RemoveContainer" containerID="71b30f7c78e9aea3a16b3c3f95cb9da3fa0c41697dcef091d1791d7487ac978f"
Sep 14 10:17:06.480929 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:06.480910    2563 scope.go:117] "RemoveContainer" containerID="71b30f7c78e9aea3a16b3c3f95cb9da3fa0c41697dcef091d1791d7487ac978f"
Sep 14 10:17:06.481233 ip-10-0-132-4 kubenswrapper[2563]: E0914 10:17:06.481212    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71b30f7c78e9aea3a16b3c3f95cb9da3fa0c41697dcef091d1791d7487ac978f\": container with ID starting with 71b30f7c78e9aea3a16b3c3f95cb9da3fa0c41697dcef091d1791d7487ac978f not found: ID does not exist" containerID="71b30f7c78e9aea3a16b3c3f95cb9da3fa0c41697dcef091d1791d7487ac978f"
Sep 14 10:17:06.481292 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:06.481243    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71b30f7c78e9aea3a16b3c3f95cb9da3fa0c41697dcef091d1791d7487ac978f"} err="failed to get container status \"71b30f7c78e9aea3a16b3c3f95cb9da3fa0c41697dcef091d1791d7487ac978f\": rpc error: code = NotFound desc = could not find container \"71b30f7c78e9aea3a16b3c3f95cb9da3fa0c41697dcef091d1791d7487ac978f\": container with ID starting with 71b30f7c78e9aea3a16b3c3f95cb9da3fa0c41697dcef091d1791d7487ac978f not found: ID does not exist"
Sep 14 10:17:06.542192 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:06.542155    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4663313e-579a-4b8d-9c26-1a10129d8eb9-dshm\") pod \"4663313e-579a-4b8d-9c26-1a10129d8eb9\" (UID: \"4663313e-579a-4b8d-9c26-1a10129d8eb9\") "
Sep 14 10:17:06.542395 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:06.542209    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4663313e-579a-4b8d-9c26-1a10129d8eb9-home\") pod \"4663313e-579a-4b8d-9c26-1a10129d8eb9\" (UID: \"4663313e-579a-4b8d-9c26-1a10129d8eb9\") "
Sep 14 10:17:06.542395 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:06.542243    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4663313e-579a-4b8d-9c26-1a10129d8eb9-tmp-dir\") pod \"4663313e-579a-4b8d-9c26-1a10129d8eb9\" (UID: \"4663313e-579a-4b8d-9c26-1a10129d8eb9\") "
Sep 14 10:17:06.542395 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:06.542267    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vdzwf\" (UniqueName: \"kubernetes.io/projected/4663313e-579a-4b8d-9c26-1a10129d8eb9-kube-api-access-vdzwf\") pod \"4663313e-579a-4b8d-9c26-1a10129d8eb9\" (UID: \"4663313e-579a-4b8d-9c26-1a10129d8eb9\") "
Sep 14 10:17:06.542395 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:06.542312    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4663313e-579a-4b8d-9c26-1a10129d8eb9-tls-certs\") pod \"4663313e-579a-4b8d-9c26-1a10129d8eb9\" (UID: \"4663313e-579a-4b8d-9c26-1a10129d8eb9\") "
Sep 14 10:17:06.542395 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:06.542339    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4663313e-579a-4b8d-9c26-1a10129d8eb9-model-cache\") pod \"4663313e-579a-4b8d-9c26-1a10129d8eb9\" (UID: \"4663313e-579a-4b8d-9c26-1a10129d8eb9\") "
Sep 14 10:17:06.542681 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:06.542569    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4663313e-579a-4b8d-9c26-1a10129d8eb9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4663313e-579a-4b8d-9c26-1a10129d8eb9" (UID: "4663313e-579a-4b8d-9c26-1a10129d8eb9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:17:06.542681 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:06.542578    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4663313e-579a-4b8d-9c26-1a10129d8eb9-home" (OuterVolumeSpecName: "home") pod "4663313e-579a-4b8d-9c26-1a10129d8eb9" (UID: "4663313e-579a-4b8d-9c26-1a10129d8eb9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:17:06.542789 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:06.542742    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4663313e-579a-4b8d-9c26-1a10129d8eb9-home\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:17:06.542789 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:06.542762    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4663313e-579a-4b8d-9c26-1a10129d8eb9-tmp-dir\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:17:06.542789 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:06.542769    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4663313e-579a-4b8d-9c26-1a10129d8eb9-model-cache" (OuterVolumeSpecName: "model-cache") pod "4663313e-579a-4b8d-9c26-1a10129d8eb9" (UID: "4663313e-579a-4b8d-9c26-1a10129d8eb9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:17:06.544652 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:06.544622    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4663313e-579a-4b8d-9c26-1a10129d8eb9-dshm" (OuterVolumeSpecName: "dshm") pod "4663313e-579a-4b8d-9c26-1a10129d8eb9" (UID: "4663313e-579a-4b8d-9c26-1a10129d8eb9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:17:06.544791 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:06.544698    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4663313e-579a-4b8d-9c26-1a10129d8eb9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4663313e-579a-4b8d-9c26-1a10129d8eb9" (UID: "4663313e-579a-4b8d-9c26-1a10129d8eb9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:17:06.544791 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:06.544728    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4663313e-579a-4b8d-9c26-1a10129d8eb9-kube-api-access-vdzwf" (OuterVolumeSpecName: "kube-api-access-vdzwf") pod "4663313e-579a-4b8d-9c26-1a10129d8eb9" (UID: "4663313e-579a-4b8d-9c26-1a10129d8eb9"). InnerVolumeSpecName "kube-api-access-vdzwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:17:06.644093 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:06.644042    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4663313e-579a-4b8d-9c26-1a10129d8eb9-dshm\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:17:06.644093 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:06.644088    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdzwf\" (UniqueName: \"kubernetes.io/projected/4663313e-579a-4b8d-9c26-1a10129d8eb9-kube-api-access-vdzwf\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:17:06.644093 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:06.644101    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4663313e-579a-4b8d-9c26-1a10129d8eb9-tls-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:17:06.644360 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:06.644115    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4663313e-579a-4b8d-9c26-1a10129d8eb9-model-cache\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:17:07.481052 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:07.481021    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-78742b/ipp-route-78742b-kserve-779f8dccff-x8bqx"
Sep 14 10:17:07.508299 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:07.508260    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-78742b/ipp-route-78742b-kserve-779f8dccff-x8bqx"]
Sep 14 10:17:07.512945 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:07.512906    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-78742b/ipp-route-78742b-kserve-779f8dccff-x8bqx"]
Sep 14 10:17:09.483677 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:09.483630    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4663313e-579a-4b8d-9c26-1a10129d8eb9" path="/var/lib/kubelet/pods/4663313e-579a-4b8d-9c26-1a10129d8eb9/volumes"
Sep 14 10:17:27.965677 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:27.965635    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:17:28.485886 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:28.485844    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:17:32.146702 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:32.146667    2563 scope.go:117] "RemoveContainer" containerID="988fe03117d1c3481af1bda1ae9c609b7599706bcc07b4c37994e86093f02fef"
Sep 14 10:17:32.156186 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:32.156161    2563 scope.go:117] "RemoveContainer" containerID="75bf44e4176adacd7e580807eda54fdcf8489562089a25e62245acc4c8a9022a"
Sep 14 10:17:34.547617 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:34.547581    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:17:34.648802 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:34.648771    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:17:34.820389 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:34.820277    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc"]
Sep 14 10:17:34.820568 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:34.820526    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc" podUID="150274d4-9512-4493-8dd0-631bcb8ce385" containerName="istio-proxy" containerID="cri-o://571caf441d66f6813ca524ce93f37f44d23ddf00f747e7a968aa54c9356c7759" gracePeriod=30
Sep 14 10:17:34.842501 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:34.842465    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:17:35.179406 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:35.179310    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2c2a39-fqkcr"]
Sep 14 10:17:35.179709 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:35.179694    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6726601c-3ca3-4150-b479-12eb34b0a0dd" containerName="istio-proxy"
Sep 14 10:17:35.179754 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:35.179710    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6726601c-3ca3-4150-b479-12eb34b0a0dd" containerName="istio-proxy"
Sep 14 10:17:35.179754 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:35.179731    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6726601c-3ca3-4150-b479-12eb34b0a0dd" containerName="istio-proxy"
Sep 14 10:17:35.179754 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:35.179736    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6726601c-3ca3-4150-b479-12eb34b0a0dd" containerName="istio-proxy"
Sep 14 10:17:35.179754 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:35.179744    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c8b7935-ed70-456d-bc61-cc4a4b0ba269" containerName="revoke-keys"
Sep 14 10:17:35.179754 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:35.179750    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c8b7935-ed70-456d-bc61-cc4a4b0ba269" containerName="revoke-keys"
Sep 14 10:17:35.179916 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:35.179757    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4663313e-579a-4b8d-9c26-1a10129d8eb9" containerName="main"
Sep 14 10:17:35.179916 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:35.179763    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4663313e-579a-4b8d-9c26-1a10129d8eb9" containerName="main"
Sep 14 10:17:35.179916 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:35.179818    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4663313e-579a-4b8d-9c26-1a10129d8eb9" containerName="main"
Sep 14 10:17:35.179916 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:35.179827    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6726601c-3ca3-4150-b479-12eb34b0a0dd" containerName="istio-proxy"
Sep 14 10:17:35.179916 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:35.179834    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6726601c-3ca3-4150-b479-12eb34b0a0dd" containerName="istio-proxy"
Sep 14 10:17:35.179916 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:35.179842    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c8b7935-ed70-456d-bc61-cc4a4b0ba269" containerName="revoke-keys"
Sep 14 10:17:35.181920 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:35.181899    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2c2a39-fqkcr"
Sep 14 10:17:35.184119 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:35.184095    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 14 10:17:35.184215 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:35.184168    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gf4xj\""
Sep 14 10:17:35.191498 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:35.191469    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2c2a39-fqkcr"]
Sep 14 10:17:35.332752 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:35.332711    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/589dd980-70cd-40a1-9295-c49c33e7a419-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2c2a39-fqkcr\" (UID: \"589dd980-70cd-40a1-9295-c49c33e7a419\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2c2a39-fqkcr"
Sep 14 10:17:35.434370 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:35.434270    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/589dd980-70cd-40a1-9295-c49c33e7a419-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2c2a39-fqkcr\" (UID: \"589dd980-70cd-40a1-9295-c49c33e7a419\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2c2a39-fqkcr"
Sep 14 10:17:35.434944 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:35.434921    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/589dd980-70cd-40a1-9295-c49c33e7a419-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2c2a39-fqkcr\" (UID: \"589dd980-70cd-40a1-9295-c49c33e7a419\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2c2a39-fqkcr"
Sep 14 10:17:35.493417 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:35.493381    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:17:35.495343 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:35.495319    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2c2a39-fqkcr"
Sep 14 10:17:35.778051 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:35.775655    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2c2a39-fqkcr"]
Sep 14 10:17:35.779183 ip-10-0-132-4 kubenswrapper[2563]: W0914 10:17:35.779149    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod589dd980_70cd_40a1_9295_c49c33e7a419.slice/crio-3cb375e86d4d22d1fe652b079509c3ddde33ab3efe93eed61ca235f2db95e0ee WatchSource:0}: Error finding container 3cb375e86d4d22d1fe652b079509c3ddde33ab3efe93eed61ca235f2db95e0ee: Status 404 returned error can't find the container with id 3cb375e86d4d22d1fe652b079509c3ddde33ab3efe93eed61ca235f2db95e0ee
Sep 14 10:17:36.269700 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:36.269660    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:17:36.661694 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:36.661600    2563 generic.go:356] "Generic (PLEG): container finished" podID="589dd980-70cd-40a1-9295-c49c33e7a419" containerID="a380450aab296c5820be9a412db508646299f50bdf8eda640bbecda11f775fd1" exitCode=0
Sep 14 10:17:36.661850 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:36.661689    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2c2a39-fqkcr" event={"ID":"589dd980-70cd-40a1-9295-c49c33e7a419","Type":"ContainerDied","Data":"a380450aab296c5820be9a412db508646299f50bdf8eda640bbecda11f775fd1"}
Sep 14 10:17:36.661850 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:36.661745    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2c2a39-fqkcr" event={"ID":"589dd980-70cd-40a1-9295-c49c33e7a419","Type":"ContainerStarted","Data":"3cb375e86d4d22d1fe652b079509c3ddde33ab3efe93eed61ca235f2db95e0ee"}
Sep 14 10:17:37.099591 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:37.099553    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:17:37.806269 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:37.806229    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2c2a39-fqkcr"
Sep 14 10:17:37.863819 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:37.863779    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:17:37.961667 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:37.961566    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/589dd980-70cd-40a1-9295-c49c33e7a419-maas-api-service-ca\") pod \"589dd980-70cd-40a1-9295-c49c33e7a419\" (UID: \"589dd980-70cd-40a1-9295-c49c33e7a419\") "
Sep 14 10:17:37.961979 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:37.961952    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/589dd980-70cd-40a1-9295-c49c33e7a419-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "589dd980-70cd-40a1-9295-c49c33e7a419" (UID: "589dd980-70cd-40a1-9295-c49c33e7a419"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:17:38.062511 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:38.062467    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/589dd980-70cd-40a1-9295-c49c33e7a419-maas-api-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:17:38.675029 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:38.674966    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2c2a39-fqkcr"
Sep 14 10:17:38.675029 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:38.675014    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2c2a39-fqkcr" event={"ID":"589dd980-70cd-40a1-9295-c49c33e7a419","Type":"ContainerDied","Data":"3cb375e86d4d22d1fe652b079509c3ddde33ab3efe93eed61ca235f2db95e0ee"}
Sep 14 10:17:38.675519 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:38.675072    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3cb375e86d4d22d1fe652b079509c3ddde33ab3efe93eed61ca235f2db95e0ee"
Sep 14 10:17:38.689580 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:38.689550    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:17:40.081926 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:40.081826    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc"
Sep 14 10:17:40.184056 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:40.183987    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/150274d4-9512-4493-8dd0-631bcb8ce385-istio-podinfo\") pod \"150274d4-9512-4493-8dd0-631bcb8ce385\" (UID: \"150274d4-9512-4493-8dd0-631bcb8ce385\") "
Sep 14 10:17:40.184056 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:40.184056    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/150274d4-9512-4493-8dd0-631bcb8ce385-istio-envoy\") pod \"150274d4-9512-4493-8dd0-631bcb8ce385\" (UID: \"150274d4-9512-4493-8dd0-631bcb8ce385\") "
Sep 14 10:17:40.184297 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:40.184077    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/150274d4-9512-4493-8dd0-631bcb8ce385-credential-socket\") pod \"150274d4-9512-4493-8dd0-631bcb8ce385\" (UID: \"150274d4-9512-4493-8dd0-631bcb8ce385\") "
Sep 14 10:17:40.184297 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:40.184157    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/150274d4-9512-4493-8dd0-631bcb8ce385-workload-socket\") pod \"150274d4-9512-4493-8dd0-631bcb8ce385\" (UID: \"150274d4-9512-4493-8dd0-631bcb8ce385\") "
Sep 14 10:17:40.184297 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:40.184184    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/150274d4-9512-4493-8dd0-631bcb8ce385-workload-certs\") pod \"150274d4-9512-4493-8dd0-631bcb8ce385\" (UID: \"150274d4-9512-4493-8dd0-631bcb8ce385\") "
Sep 14 10:17:40.184297 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:40.184199    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/150274d4-9512-4493-8dd0-631bcb8ce385-istio-data\") pod \"150274d4-9512-4493-8dd0-631bcb8ce385\" (UID: \"150274d4-9512-4493-8dd0-631bcb8ce385\") "
Sep 14 10:17:40.184297 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:40.184230    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/150274d4-9512-4493-8dd0-631bcb8ce385-istio-token\") pod \"150274d4-9512-4493-8dd0-631bcb8ce385\" (UID: \"150274d4-9512-4493-8dd0-631bcb8ce385\") "
Sep 14 10:17:40.184297 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:40.184258    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/150274d4-9512-4493-8dd0-631bcb8ce385-istiod-ca-cert\") pod \"150274d4-9512-4493-8dd0-631bcb8ce385\" (UID: \"150274d4-9512-4493-8dd0-631bcb8ce385\") "
Sep 14 10:17:40.184297 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:40.184282    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8ds2d\" (UniqueName: \"kubernetes.io/projected/150274d4-9512-4493-8dd0-631bcb8ce385-kube-api-access-8ds2d\") pod \"150274d4-9512-4493-8dd0-631bcb8ce385\" (UID: \"150274d4-9512-4493-8dd0-631bcb8ce385\") "
Sep 14 10:17:40.184667 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:40.184447    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/150274d4-9512-4493-8dd0-631bcb8ce385-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "150274d4-9512-4493-8dd0-631bcb8ce385" (UID: "150274d4-9512-4493-8dd0-631bcb8ce385"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:17:40.184667 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:40.184510    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/150274d4-9512-4493-8dd0-631bcb8ce385-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "150274d4-9512-4493-8dd0-631bcb8ce385" (UID: "150274d4-9512-4493-8dd0-631bcb8ce385"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:17:40.184667 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:40.184521    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/150274d4-9512-4493-8dd0-631bcb8ce385-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "150274d4-9512-4493-8dd0-631bcb8ce385" (UID: "150274d4-9512-4493-8dd0-631bcb8ce385"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:17:40.184841 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:40.184695    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/150274d4-9512-4493-8dd0-631bcb8ce385-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:17:40.184841 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:40.184714    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/150274d4-9512-4493-8dd0-631bcb8ce385-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:17:40.184841 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:40.184728    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/150274d4-9512-4493-8dd0-631bcb8ce385-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:17:40.184841 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:40.184825    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/150274d4-9512-4493-8dd0-631bcb8ce385-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "150274d4-9512-4493-8dd0-631bcb8ce385" (UID: "150274d4-9512-4493-8dd0-631bcb8ce385"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:17:40.185079 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:40.184848    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/150274d4-9512-4493-8dd0-631bcb8ce385-istio-data" (OuterVolumeSpecName: "istio-data") pod "150274d4-9512-4493-8dd0-631bcb8ce385" (UID: "150274d4-9512-4493-8dd0-631bcb8ce385"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:17:40.186444 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:40.186399    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/150274d4-9512-4493-8dd0-631bcb8ce385-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "150274d4-9512-4493-8dd0-631bcb8ce385" (UID: "150274d4-9512-4493-8dd0-631bcb8ce385"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:17:40.186573 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:40.186499    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/150274d4-9512-4493-8dd0-631bcb8ce385-istio-token" (OuterVolumeSpecName: "istio-token") pod "150274d4-9512-4493-8dd0-631bcb8ce385" (UID: "150274d4-9512-4493-8dd0-631bcb8ce385"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:17:40.186754 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:40.186724    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/150274d4-9512-4493-8dd0-631bcb8ce385-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "150274d4-9512-4493-8dd0-631bcb8ce385" (UID: "150274d4-9512-4493-8dd0-631bcb8ce385"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 10:17:40.186754 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:40.186747    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/150274d4-9512-4493-8dd0-631bcb8ce385-kube-api-access-8ds2d" (OuterVolumeSpecName: "kube-api-access-8ds2d") pod "150274d4-9512-4493-8dd0-631bcb8ce385" (UID: "150274d4-9512-4493-8dd0-631bcb8ce385"). InnerVolumeSpecName "kube-api-access-8ds2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:17:40.286255 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:40.286208    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/150274d4-9512-4493-8dd0-631bcb8ce385-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:17:40.286255 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:40.286243    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/150274d4-9512-4493-8dd0-631bcb8ce385-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:17:40.286255 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:40.286252    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/150274d4-9512-4493-8dd0-631bcb8ce385-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:17:40.286255 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:40.286264    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ds2d\" (UniqueName: \"kubernetes.io/projected/150274d4-9512-4493-8dd0-631bcb8ce385-kube-api-access-8ds2d\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:17:40.286553 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:40.286275    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/150274d4-9512-4493-8dd0-631bcb8ce385-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:17:40.286553 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:40.286284    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/150274d4-9512-4493-8dd0-631bcb8ce385-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:17:40.688948 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:40.688915    2563 generic.go:356] "Generic (PLEG): container finished" podID="150274d4-9512-4493-8dd0-631bcb8ce385" containerID="571caf441d66f6813ca524ce93f37f44d23ddf00f747e7a968aa54c9356c7759" exitCode=0
Sep 14 10:17:40.689145 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:40.688966    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc"
Sep 14 10:17:40.689145 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:40.689026    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc" event={"ID":"150274d4-9512-4493-8dd0-631bcb8ce385","Type":"ContainerDied","Data":"571caf441d66f6813ca524ce93f37f44d23ddf00f747e7a968aa54c9356c7759"}
Sep 14 10:17:40.689145 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:40.689092    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc" event={"ID":"150274d4-9512-4493-8dd0-631bcb8ce385","Type":"ContainerDied","Data":"2fa53f4a74ccfe6615e482730cfb706ee4b00ea1fa707e7ef091361057484a04"}
Sep 14 10:17:40.689145 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:40.689121    2563 scope.go:117] "RemoveContainer" containerID="571caf441d66f6813ca524ce93f37f44d23ddf00f747e7a968aa54c9356c7759"
Sep 14 10:17:40.699820 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:40.699800    2563 scope.go:117] "RemoveContainer" containerID="ddc620850fc188aa493d9995584c4c58e711f0fb47bba7541cbf86787590383c"
Sep 14 10:17:40.710244 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:40.710219    2563 scope.go:117] "RemoveContainer" containerID="571caf441d66f6813ca524ce93f37f44d23ddf00f747e7a968aa54c9356c7759"
Sep 14 10:17:40.710639 ip-10-0-132-4 kubenswrapper[2563]: E0914 10:17:40.710614    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"571caf441d66f6813ca524ce93f37f44d23ddf00f747e7a968aa54c9356c7759\": container with ID starting with 571caf441d66f6813ca524ce93f37f44d23ddf00f747e7a968aa54c9356c7759 not found: ID does not exist" containerID="571caf441d66f6813ca524ce93f37f44d23ddf00f747e7a968aa54c9356c7759"
Sep 14 10:17:40.710719 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:40.710649    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"571caf441d66f6813ca524ce93f37f44d23ddf00f747e7a968aa54c9356c7759"} err="failed to get container status \"571caf441d66f6813ca524ce93f37f44d23ddf00f747e7a968aa54c9356c7759\": rpc error: code = NotFound desc = could not find container \"571caf441d66f6813ca524ce93f37f44d23ddf00f747e7a968aa54c9356c7759\": container with ID starting with 571caf441d66f6813ca524ce93f37f44d23ddf00f747e7a968aa54c9356c7759 not found: ID does not exist"
Sep 14 10:17:40.710719 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:40.710669    2563 scope.go:117] "RemoveContainer" containerID="ddc620850fc188aa493d9995584c4c58e711f0fb47bba7541cbf86787590383c"
Sep 14 10:17:40.710966 ip-10-0-132-4 kubenswrapper[2563]: E0914 10:17:40.710947    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddc620850fc188aa493d9995584c4c58e711f0fb47bba7541cbf86787590383c\": container with ID starting with ddc620850fc188aa493d9995584c4c58e711f0fb47bba7541cbf86787590383c not found: ID does not exist" containerID="ddc620850fc188aa493d9995584c4c58e711f0fb47bba7541cbf86787590383c"
Sep 14 10:17:40.711034 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:40.710977    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddc620850fc188aa493d9995584c4c58e711f0fb47bba7541cbf86787590383c"} err="failed to get container status \"ddc620850fc188aa493d9995584c4c58e711f0fb47bba7541cbf86787590383c\": rpc error: code = NotFound desc = could not find container \"ddc620850fc188aa493d9995584c4c58e711f0fb47bba7541cbf86787590383c\": container with ID starting with ddc620850fc188aa493d9995584c4c58e711f0fb47bba7541cbf86787590383c not found: ID does not exist"
Sep 14 10:17:40.718618 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:40.718571    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc"]
Sep 14 10:17:40.726902 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:40.726864    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-78742b-openshift-default-65b94d7446-fm6vc"]
Sep 14 10:17:41.482967 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:41.482923    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="150274d4-9512-4493-8dd0-631bcb8ce385" path="/var/lib/kubelet/pods/150274d4-9512-4493-8dd0-631bcb8ce385/volumes"
Sep 14 10:17:45.814169 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:45.814117    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2c2a39-29823015-5q59k"]
Sep 14 10:17:45.816806 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:45.816774    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2c2a39-29823015-5q59k"]
Sep 14 10:17:45.817734 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:45.817710    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:17:45.915408 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:45.915373    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:17:46.029221 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:46.029186    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:17:46.080371 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:46.080275    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2c2a39-fqkcr"]
Sep 14 10:17:46.082885 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:46.082854    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2c2a39-fqkcr"]
Sep 14 10:17:46.107395 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:46.107365    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:17:46.445947 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:46.445856    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:17:46.607420 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:46.607390    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:17:47.482687 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:47.482649    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="589dd980-70cd-40a1-9295-c49c33e7a419" path="/var/lib/kubelet/pods/589dd980-70cd-40a1-9295-c49c33e7a419/volumes"
Sep 14 10:17:47.483112 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:47.482956    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6683dc2-8a3f-40c9-8b86-0045280e96e1" path="/var/lib/kubelet/pods/e6683dc2-8a3f-40c9-8b86-0045280e96e1/volumes"
Sep 14 10:17:47.522194 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:47.522159    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:17:48.300666 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:48.300632    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:17:52.576662 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:52.576621    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:17:52.700582 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:52.700540    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:17:52.825152 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:52.825105    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n"]
Sep 14 10:17:52.825399 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:52.825374    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n" podUID="51e15aea-5153-46bc-b34c-e186355d15f6" containerName="istio-proxy" containerID="cri-o://89454800b3d11bd90e0eaaeaa4a2637f6cd3464efc29df3ea976d2959f7f2e3f" gracePeriod=30
Sep 14 10:17:52.841713 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:52.841627    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:17:53.162130 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:53.162031    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-51aefd-c6qsr"]
Sep 14 10:17:53.162421 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:53.162407    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="150274d4-9512-4493-8dd0-631bcb8ce385" containerName="istio-proxy"
Sep 14 10:17:53.162470 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:53.162423    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="150274d4-9512-4493-8dd0-631bcb8ce385" containerName="istio-proxy"
Sep 14 10:17:53.162470 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:53.162431    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="150274d4-9512-4493-8dd0-631bcb8ce385" containerName="istio-proxy"
Sep 14 10:17:53.162470 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:53.162437    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="150274d4-9512-4493-8dd0-631bcb8ce385" containerName="istio-proxy"
Sep 14 10:17:53.162470 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:53.162464    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="589dd980-70cd-40a1-9295-c49c33e7a419" containerName="revoke-keys"
Sep 14 10:17:53.162590 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:53.162471    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="589dd980-70cd-40a1-9295-c49c33e7a419" containerName="revoke-keys"
Sep 14 10:17:53.162590 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:53.162526    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="150274d4-9512-4493-8dd0-631bcb8ce385" containerName="istio-proxy"
Sep 14 10:17:53.162590 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:53.162536    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="150274d4-9512-4493-8dd0-631bcb8ce385" containerName="istio-proxy"
Sep 14 10:17:53.162590 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:53.162543    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="589dd980-70cd-40a1-9295-c49c33e7a419" containerName="revoke-keys"
Sep 14 10:17:53.167198 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:53.167173    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-51aefd-c6qsr"
Sep 14 10:17:53.169667 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:53.169635    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 14 10:17:53.169852 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:53.169681    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gf4xj\""
Sep 14 10:17:53.174685 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:53.174643    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-51aefd-c6qsr"]
Sep 14 10:17:53.209713 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:53.209672    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2784d964-c759-4d84-8fd8-99ad6664c4da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-51aefd-c6qsr\" (UID: \"2784d964-c759-4d84-8fd8-99ad6664c4da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-51aefd-c6qsr"
Sep 14 10:17:53.310994 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:53.310886    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2784d964-c759-4d84-8fd8-99ad6664c4da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-51aefd-c6qsr\" (UID: \"2784d964-c759-4d84-8fd8-99ad6664c4da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-51aefd-c6qsr"
Sep 14 10:17:53.311590 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:53.311565    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2784d964-c759-4d84-8fd8-99ad6664c4da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-51aefd-c6qsr\" (UID: \"2784d964-c759-4d84-8fd8-99ad6664c4da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-51aefd-c6qsr"
Sep 14 10:17:53.376407 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:53.376368    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:17:53.482600 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:53.482491    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-51aefd-c6qsr"
Sep 14 10:17:53.758799 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:53.758769    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-51aefd-c6qsr"]
Sep 14 10:17:53.771496 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:53.771469    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-51aefd-c6qsr" event={"ID":"2784d964-c759-4d84-8fd8-99ad6664c4da","Type":"ContainerStarted","Data":"ae7044388d4147fc7d2b09b510b3ed51e304210984d7fd0dd53054d17c8302d6"}
Sep 14 10:17:54.213719 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:54.213683    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:17:54.779167 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:54.779134    2563 generic.go:356] "Generic (PLEG): container finished" podID="2784d964-c759-4d84-8fd8-99ad6664c4da" containerID="b5dee3a37aa7fac726948526f664978c30844916d916d30fcf9795fc0e57e53b" exitCode=0
Sep 14 10:17:54.779686 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:54.779193    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-51aefd-c6qsr" event={"ID":"2784d964-c759-4d84-8fd8-99ad6664c4da","Type":"ContainerDied","Data":"b5dee3a37aa7fac726948526f664978c30844916d916d30fcf9795fc0e57e53b"}
Sep 14 10:17:55.011438 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:55.011404    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:17:55.765481 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:55.765447    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:17:55.931655 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:55.931616    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-51aefd-c6qsr"
Sep 14 10:17:56.036038 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:56.035904    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2784d964-c759-4d84-8fd8-99ad6664c4da-maas-api-service-ca\") pod \"2784d964-c759-4d84-8fd8-99ad6664c4da\" (UID: \"2784d964-c759-4d84-8fd8-99ad6664c4da\") "
Sep 14 10:17:56.036349 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:56.036323    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2784d964-c759-4d84-8fd8-99ad6664c4da-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2784d964-c759-4d84-8fd8-99ad6664c4da" (UID: "2784d964-c759-4d84-8fd8-99ad6664c4da"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:17:56.136967 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:56.136918    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2784d964-c759-4d84-8fd8-99ad6664c4da-maas-api-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:17:56.794268 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:56.794238    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-51aefd-c6qsr"
Sep 14 10:17:56.794480 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:56.794244    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-51aefd-c6qsr" event={"ID":"2784d964-c759-4d84-8fd8-99ad6664c4da","Type":"ContainerDied","Data":"ae7044388d4147fc7d2b09b510b3ed51e304210984d7fd0dd53054d17c8302d6"}
Sep 14 10:17:56.794480 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:56.794383    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae7044388d4147fc7d2b09b510b3ed51e304210984d7fd0dd53054d17c8302d6"
Sep 14 10:17:57.310718 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:57.310677    2563 patch_prober.go:28] interesting pod/e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 14 10:17:57.311132 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:57.310746    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n" podUID="51e15aea-5153-46bc-b34c-e186355d15f6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 14 10:17:58.090252 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:58.090168    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n"
Sep 14 10:17:58.157741 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:58.157702    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51e15aea-5153-46bc-b34c-e186355d15f6-istiod-ca-cert\") pod \"51e15aea-5153-46bc-b34c-e186355d15f6\" (UID: \"51e15aea-5153-46bc-b34c-e186355d15f6\") "
Sep 14 10:17:58.157960 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:58.157761    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4jdng\" (UniqueName: \"kubernetes.io/projected/51e15aea-5153-46bc-b34c-e186355d15f6-kube-api-access-4jdng\") pod \"51e15aea-5153-46bc-b34c-e186355d15f6\" (UID: \"51e15aea-5153-46bc-b34c-e186355d15f6\") "
Sep 14 10:17:58.157960 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:58.157809    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51e15aea-5153-46bc-b34c-e186355d15f6-istio-data\") pod \"51e15aea-5153-46bc-b34c-e186355d15f6\" (UID: \"51e15aea-5153-46bc-b34c-e186355d15f6\") "
Sep 14 10:17:58.157960 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:58.157870    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51e15aea-5153-46bc-b34c-e186355d15f6-workload-certs\") pod \"51e15aea-5153-46bc-b34c-e186355d15f6\" (UID: \"51e15aea-5153-46bc-b34c-e186355d15f6\") "
Sep 14 10:17:58.158188 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:58.158028    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51e15aea-5153-46bc-b34c-e186355d15f6-istio-token\") pod \"51e15aea-5153-46bc-b34c-e186355d15f6\" (UID: \"51e15aea-5153-46bc-b34c-e186355d15f6\") "
Sep 14 10:17:58.158188 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:58.158130    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51e15aea-5153-46bc-b34c-e186355d15f6-workload-socket\") pod \"51e15aea-5153-46bc-b34c-e186355d15f6\" (UID: \"51e15aea-5153-46bc-b34c-e186355d15f6\") "
Sep 14 10:17:58.158188 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:58.158185    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51e15aea-5153-46bc-b34c-e186355d15f6-istio-podinfo\") pod \"51e15aea-5153-46bc-b34c-e186355d15f6\" (UID: \"51e15aea-5153-46bc-b34c-e186355d15f6\") "
Sep 14 10:17:58.158340 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:58.158211    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51e15aea-5153-46bc-b34c-e186355d15f6-credential-socket\") pod \"51e15aea-5153-46bc-b34c-e186355d15f6\" (UID: \"51e15aea-5153-46bc-b34c-e186355d15f6\") "
Sep 14 10:17:58.158340 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:58.158220    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51e15aea-5153-46bc-b34c-e186355d15f6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "51e15aea-5153-46bc-b34c-e186355d15f6" (UID: "51e15aea-5153-46bc-b34c-e186355d15f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:17:58.158340 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:58.158231    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51e15aea-5153-46bc-b34c-e186355d15f6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "51e15aea-5153-46bc-b34c-e186355d15f6" (UID: "51e15aea-5153-46bc-b34c-e186355d15f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:17:58.158340 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:58.158216    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51e15aea-5153-46bc-b34c-e186355d15f6-istio-data" (OuterVolumeSpecName: "istio-data") pod "51e15aea-5153-46bc-b34c-e186355d15f6" (UID: "51e15aea-5153-46bc-b34c-e186355d15f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:17:58.158340 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:58.158240    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51e15aea-5153-46bc-b34c-e186355d15f6-istio-envoy\") pod \"51e15aea-5153-46bc-b34c-e186355d15f6\" (UID: \"51e15aea-5153-46bc-b34c-e186355d15f6\") "
Sep 14 10:17:58.158663 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:58.158638    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51e15aea-5153-46bc-b34c-e186355d15f6-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:17:58.158806 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:58.158662    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51e15aea-5153-46bc-b34c-e186355d15f6-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:17:58.158806 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:58.158693    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51e15aea-5153-46bc-b34c-e186355d15f6-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:17:58.158806 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:58.158764    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51e15aea-5153-46bc-b34c-e186355d15f6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "51e15aea-5153-46bc-b34c-e186355d15f6" (UID: "51e15aea-5153-46bc-b34c-e186355d15f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:17:58.158988 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:58.158819    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51e15aea-5153-46bc-b34c-e186355d15f6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "51e15aea-5153-46bc-b34c-e186355d15f6" (UID: "51e15aea-5153-46bc-b34c-e186355d15f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:17:58.160464 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:58.160433    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51e15aea-5153-46bc-b34c-e186355d15f6-istio-token" (OuterVolumeSpecName: "istio-token") pod "51e15aea-5153-46bc-b34c-e186355d15f6" (UID: "51e15aea-5153-46bc-b34c-e186355d15f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:17:58.160598 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:58.160459    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51e15aea-5153-46bc-b34c-e186355d15f6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "51e15aea-5153-46bc-b34c-e186355d15f6" (UID: "51e15aea-5153-46bc-b34c-e186355d15f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 10:17:58.160660 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:58.160604    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51e15aea-5153-46bc-b34c-e186355d15f6-kube-api-access-4jdng" (OuterVolumeSpecName: "kube-api-access-4jdng") pod "51e15aea-5153-46bc-b34c-e186355d15f6" (UID: "51e15aea-5153-46bc-b34c-e186355d15f6"). InnerVolumeSpecName "kube-api-access-4jdng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:17:58.160660 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:58.160628    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51e15aea-5153-46bc-b34c-e186355d15f6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "51e15aea-5153-46bc-b34c-e186355d15f6" (UID: "51e15aea-5153-46bc-b34c-e186355d15f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:17:58.259653 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:58.259608    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51e15aea-5153-46bc-b34c-e186355d15f6-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:17:58.259653 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:58.259643    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51e15aea-5153-46bc-b34c-e186355d15f6-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:17:58.259653 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:58.259656    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51e15aea-5153-46bc-b34c-e186355d15f6-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:17:58.259902 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:58.259667    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51e15aea-5153-46bc-b34c-e186355d15f6-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:17:58.259902 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:58.259677    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jdng\" (UniqueName: \"kubernetes.io/projected/51e15aea-5153-46bc-b34c-e186355d15f6-kube-api-access-4jdng\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:17:58.259902 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:58.259689    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51e15aea-5153-46bc-b34c-e186355d15f6-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:17:58.808310 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:58.808273    2563 generic.go:356] "Generic (PLEG): container finished" podID="51e15aea-5153-46bc-b34c-e186355d15f6" containerID="89454800b3d11bd90e0eaaeaa4a2637f6cd3464efc29df3ea976d2959f7f2e3f" exitCode=0
Sep 14 10:17:58.808746 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:58.808318    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n" event={"ID":"51e15aea-5153-46bc-b34c-e186355d15f6","Type":"ContainerDied","Data":"89454800b3d11bd90e0eaaeaa4a2637f6cd3464efc29df3ea976d2959f7f2e3f"}
Sep 14 10:17:58.808746 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:58.808330    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n"
Sep 14 10:17:58.808746 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:58.808371    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n" event={"ID":"51e15aea-5153-46bc-b34c-e186355d15f6","Type":"ContainerDied","Data":"d766e62891caef2abbc564ddfd909554d01b7749c0adc46b425f2d3052e5fb72"}
Sep 14 10:17:58.808746 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:58.808397    2563 scope.go:117] "RemoveContainer" containerID="89454800b3d11bd90e0eaaeaa4a2637f6cd3464efc29df3ea976d2959f7f2e3f"
Sep 14 10:17:58.819184 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:58.819154    2563 scope.go:117] "RemoveContainer" containerID="90c883ed9147b08e0ee074058baffc8bdfdd525132bd387c0c32b1a9933471a8"
Sep 14 10:17:58.829384 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:58.829355    2563 scope.go:117] "RemoveContainer" containerID="89454800b3d11bd90e0eaaeaa4a2637f6cd3464efc29df3ea976d2959f7f2e3f"
Sep 14 10:17:58.829759 ip-10-0-132-4 kubenswrapper[2563]: E0914 10:17:58.829738    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89454800b3d11bd90e0eaaeaa4a2637f6cd3464efc29df3ea976d2959f7f2e3f\": container with ID starting with 89454800b3d11bd90e0eaaeaa4a2637f6cd3464efc29df3ea976d2959f7f2e3f not found: ID does not exist" containerID="89454800b3d11bd90e0eaaeaa4a2637f6cd3464efc29df3ea976d2959f7f2e3f"
Sep 14 10:17:58.829820 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:58.829771    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89454800b3d11bd90e0eaaeaa4a2637f6cd3464efc29df3ea976d2959f7f2e3f"} err="failed to get container status \"89454800b3d11bd90e0eaaeaa4a2637f6cd3464efc29df3ea976d2959f7f2e3f\": rpc error: code = NotFound desc = could not find container \"89454800b3d11bd90e0eaaeaa4a2637f6cd3464efc29df3ea976d2959f7f2e3f\": container with ID starting with 89454800b3d11bd90e0eaaeaa4a2637f6cd3464efc29df3ea976d2959f7f2e3f not found: ID does not exist"
Sep 14 10:17:58.829820 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:58.829791    2563 scope.go:117] "RemoveContainer" containerID="90c883ed9147b08e0ee074058baffc8bdfdd525132bd387c0c32b1a9933471a8"
Sep 14 10:17:58.830170 ip-10-0-132-4 kubenswrapper[2563]: E0914 10:17:58.830145    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90c883ed9147b08e0ee074058baffc8bdfdd525132bd387c0c32b1a9933471a8\": container with ID starting with 90c883ed9147b08e0ee074058baffc8bdfdd525132bd387c0c32b1a9933471a8 not found: ID does not exist" containerID="90c883ed9147b08e0ee074058baffc8bdfdd525132bd387c0c32b1a9933471a8"
Sep 14 10:17:58.830233 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:58.830176    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90c883ed9147b08e0ee074058baffc8bdfdd525132bd387c0c32b1a9933471a8"} err="failed to get container status \"90c883ed9147b08e0ee074058baffc8bdfdd525132bd387c0c32b1a9933471a8\": rpc error: code = NotFound desc = could not find container \"90c883ed9147b08e0ee074058baffc8bdfdd525132bd387c0c32b1a9933471a8\": container with ID starting with 90c883ed9147b08e0ee074058baffc8bdfdd525132bd387c0c32b1a9933471a8 not found: ID does not exist"
Sep 14 10:17:58.835852 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:58.835812    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n"]
Sep 14 10:17:58.839873 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:58.839836    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-2c2a39-openshift-default-58ffb648bd-qqt4n"]
Sep 14 10:17:59.482846 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:17:59.482808    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51e15aea-5153-46bc-b34c-e186355d15f6" path="/var/lib/kubelet/pods/51e15aea-5153-46bc-b34c-e186355d15f6/volumes"
Sep 14 10:18:03.805785 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:03.805750    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:18:03.892246 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:03.892181    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:18:03.989539 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:03.989500    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:18:04.048033 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:04.047969    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-51aefd-c6qsr"]
Sep 14 10:18:04.050579 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:04.050544    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-51aefd-c6qsr"]
Sep 14 10:18:04.092575 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:04.092481    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:18:04.435502 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:04.435414    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:18:04.535334 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:04.535302    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:18:05.395470 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:05.395437    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:18:05.482459 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:05.482422    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2784d964-c759-4d84-8fd8-99ad6664c4da" path="/var/lib/kubelet/pods/2784d964-c759-4d84-8fd8-99ad6664c4da/volumes"
Sep 14 10:18:10.524411 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:10.524372    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:18:10.627327 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:10.627290    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:18:10.751549 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:10.751504    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp"]
Sep 14 10:18:10.751796 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:10.751774    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp" podUID="e1bb4690-068b-43c3-abdb-8bcda1591ae4" containerName="istio-proxy" containerID="cri-o://db6fbf4ee39a7355080a64b101402f12d2fcf8c529d7b5256ba8b0d40ef5d2a5" gracePeriod=30
Sep 14 10:18:10.778382 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:10.778284    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:18:11.429209 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:11.429181    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:18:12.333368 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:12.333315    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:18:13.189114 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:13.189066    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:18:15.917276 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:15.917234    2563 generic.go:356] "Generic (PLEG): container finished" podID="e1bb4690-068b-43c3-abdb-8bcda1591ae4" containerID="db6fbf4ee39a7355080a64b101402f12d2fcf8c529d7b5256ba8b0d40ef5d2a5" exitCode=0
Sep 14 10:18:15.917276 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:15.917267    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp" event={"ID":"e1bb4690-068b-43c3-abdb-8bcda1591ae4","Type":"ContainerDied","Data":"db6fbf4ee39a7355080a64b101402f12d2fcf8c529d7b5256ba8b0d40ef5d2a5"}
Sep 14 10:18:15.917696 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:15.917339    2563 scope.go:117] "RemoveContainer" containerID="2f244eaaaa673331a39611fe59a0f1a394dab63f034769c86e833c709e2d4aaa"
Sep 14 10:18:16.019869 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:16.019785    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp"
Sep 14 10:18:16.123919 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:16.123885    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1bb4690-068b-43c3-abdb-8bcda1591ae4-istio-podinfo\") pod \"e1bb4690-068b-43c3-abdb-8bcda1591ae4\" (UID: \"e1bb4690-068b-43c3-abdb-8bcda1591ae4\") "
Sep 14 10:18:16.124124 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:16.123938    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cvgn8\" (UniqueName: \"kubernetes.io/projected/e1bb4690-068b-43c3-abdb-8bcda1591ae4-kube-api-access-cvgn8\") pod \"e1bb4690-068b-43c3-abdb-8bcda1591ae4\" (UID: \"e1bb4690-068b-43c3-abdb-8bcda1591ae4\") "
Sep 14 10:18:16.124186 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:16.124130    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1bb4690-068b-43c3-abdb-8bcda1591ae4-istiod-ca-cert\") pod \"e1bb4690-068b-43c3-abdb-8bcda1591ae4\" (UID: \"e1bb4690-068b-43c3-abdb-8bcda1591ae4\") "
Sep 14 10:18:16.124186 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:16.124165    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1bb4690-068b-43c3-abdb-8bcda1591ae4-workload-socket\") pod \"e1bb4690-068b-43c3-abdb-8bcda1591ae4\" (UID: \"e1bb4690-068b-43c3-abdb-8bcda1591ae4\") "
Sep 14 10:18:16.124307 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:16.124187    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1bb4690-068b-43c3-abdb-8bcda1591ae4-istio-data\") pod \"e1bb4690-068b-43c3-abdb-8bcda1591ae4\" (UID: \"e1bb4690-068b-43c3-abdb-8bcda1591ae4\") "
Sep 14 10:18:16.124307 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:16.124218    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1bb4690-068b-43c3-abdb-8bcda1591ae4-workload-certs\") pod \"e1bb4690-068b-43c3-abdb-8bcda1591ae4\" (UID: \"e1bb4690-068b-43c3-abdb-8bcda1591ae4\") "
Sep 14 10:18:16.124307 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:16.124248    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1bb4690-068b-43c3-abdb-8bcda1591ae4-istio-envoy\") pod \"e1bb4690-068b-43c3-abdb-8bcda1591ae4\" (UID: \"e1bb4690-068b-43c3-abdb-8bcda1591ae4\") "
Sep 14 10:18:16.124307 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:16.124293    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1bb4690-068b-43c3-abdb-8bcda1591ae4-credential-socket\") pod \"e1bb4690-068b-43c3-abdb-8bcda1591ae4\" (UID: \"e1bb4690-068b-43c3-abdb-8bcda1591ae4\") "
Sep 14 10:18:16.124525 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:16.124326    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1bb4690-068b-43c3-abdb-8bcda1591ae4-istio-token\") pod \"e1bb4690-068b-43c3-abdb-8bcda1591ae4\" (UID: \"e1bb4690-068b-43c3-abdb-8bcda1591ae4\") "
Sep 14 10:18:16.124525 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:16.124495    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1bb4690-068b-43c3-abdb-8bcda1591ae4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e1bb4690-068b-43c3-abdb-8bcda1591ae4" (UID: "e1bb4690-068b-43c3-abdb-8bcda1591ae4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:18:16.124637 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:16.124538    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1bb4690-068b-43c3-abdb-8bcda1591ae4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e1bb4690-068b-43c3-abdb-8bcda1591ae4" (UID: "e1bb4690-068b-43c3-abdb-8bcda1591ae4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:18:16.124637 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:16.124598    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1bb4690-068b-43c3-abdb-8bcda1591ae4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e1bb4690-068b-43c3-abdb-8bcda1591ae4" (UID: "e1bb4690-068b-43c3-abdb-8bcda1591ae4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:18:16.124746 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:16.124678    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1bb4690-068b-43c3-abdb-8bcda1591ae4-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:18:16.124746 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:16.124698    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1bb4690-068b-43c3-abdb-8bcda1591ae4-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:18:16.124746 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:16.124712    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1bb4690-068b-43c3-abdb-8bcda1591ae4-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:18:16.124746 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:16.124714    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1bb4690-068b-43c3-abdb-8bcda1591ae4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e1bb4690-068b-43c3-abdb-8bcda1591ae4" (UID: "e1bb4690-068b-43c3-abdb-8bcda1591ae4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:18:16.124885 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:16.124738    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1bb4690-068b-43c3-abdb-8bcda1591ae4-istio-data" (OuterVolumeSpecName: "istio-data") pod "e1bb4690-068b-43c3-abdb-8bcda1591ae4" (UID: "e1bb4690-068b-43c3-abdb-8bcda1591ae4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:18:16.126379 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:16.126345    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e1bb4690-068b-43c3-abdb-8bcda1591ae4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e1bb4690-068b-43c3-abdb-8bcda1591ae4" (UID: "e1bb4690-068b-43c3-abdb-8bcda1591ae4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 10:18:16.126379 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:16.126359    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1bb4690-068b-43c3-abdb-8bcda1591ae4-kube-api-access-cvgn8" (OuterVolumeSpecName: "kube-api-access-cvgn8") pod "e1bb4690-068b-43c3-abdb-8bcda1591ae4" (UID: "e1bb4690-068b-43c3-abdb-8bcda1591ae4"). InnerVolumeSpecName "kube-api-access-cvgn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:18:16.126557 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:16.126487    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1bb4690-068b-43c3-abdb-8bcda1591ae4-istio-token" (OuterVolumeSpecName: "istio-token") pod "e1bb4690-068b-43c3-abdb-8bcda1591ae4" (UID: "e1bb4690-068b-43c3-abdb-8bcda1591ae4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:18:16.126615 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:16.126590    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1bb4690-068b-43c3-abdb-8bcda1591ae4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e1bb4690-068b-43c3-abdb-8bcda1591ae4" (UID: "e1bb4690-068b-43c3-abdb-8bcda1591ae4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:18:16.226076 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:16.226035    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1bb4690-068b-43c3-abdb-8bcda1591ae4-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:18:16.226076 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:16.226067    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1bb4690-068b-43c3-abdb-8bcda1591ae4-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:18:16.226076 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:16.226077    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1bb4690-068b-43c3-abdb-8bcda1591ae4-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:18:16.226076 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:16.226088    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvgn8\" (UniqueName: \"kubernetes.io/projected/e1bb4690-068b-43c3-abdb-8bcda1591ae4-kube-api-access-cvgn8\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:18:16.226369 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:16.226097    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1bb4690-068b-43c3-abdb-8bcda1591ae4-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:18:16.226369 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:16.226106    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1bb4690-068b-43c3-abdb-8bcda1591ae4-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:18:16.925553 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:16.925522    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp"
Sep 14 10:18:16.926036 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:16.925570    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp" event={"ID":"e1bb4690-068b-43c3-abdb-8bcda1591ae4","Type":"ContainerDied","Data":"81428c8f44c6225ed5125de90e4955519a373816313d58488b943214f74fafb2"}
Sep 14 10:18:16.926036 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:16.925635    2563 scope.go:117] "RemoveContainer" containerID="db6fbf4ee39a7355080a64b101402f12d2fcf8c529d7b5256ba8b0d40ef5d2a5"
Sep 14 10:18:16.949243 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:16.949200    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp"]
Sep 14 10:18:16.952872 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:16.952836    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-51aefd-openshift-default-586b7d886d-m5dlp"]
Sep 14 10:18:17.482351 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:17.482314    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1bb4690-068b-43c3-abdb-8bcda1591ae4" path="/var/lib/kubelet/pods/e1bb4690-068b-43c3-abdb-8bcda1591ae4/volumes"
Sep 14 10:18:21.696655 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:21.696610    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f4d054-29823015-64dc9"]
Sep 14 10:18:21.700283 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:21.700247    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:18:21.701025 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:21.700983    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f4d054-29823015-64dc9"]
Sep 14 10:18:21.805712 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:21.805672    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:18:21.901439 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:21.901380    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:18:21.963448 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:21.963362    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:18:22.277361 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:22.277328    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:18:22.426810 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:22.426776    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:18:23.483216 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:23.483175    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be625209-24e3-4c43-8595-2cf4842a92a0" path="/var/lib/kubelet/pods/be625209-24e3-4c43-8595-2cf4842a92a0/volumes"
Sep 14 10:18:28.547176 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:28.547140    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:18:28.614571 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:28.614539    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:18:28.846917 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:28.846816    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl"]
Sep 14 10:18:28.847130 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:28.847049    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl" podUID="3d80c24f-867f-4c36-9e8b-3ab262ef1fb7" containerName="istio-proxy" containerID="cri-o://6404f8bf1af89d10c30c8c85d005419ef6e72c50ef56a42c9636d841c9dd3813" gracePeriod=30
Sep 14 10:18:28.882656 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:28.882617    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:18:31.300793 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:31.300753    2563 patch_prober.go:28] interesting pod/e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 14 10:18:31.301280 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:31.300835    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl" podUID="3d80c24f-867f-4c36-9e8b-3ab262ef1fb7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 14 10:18:34.029952 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:34.029916    2563 generic.go:356] "Generic (PLEG): container finished" podID="3d80c24f-867f-4c36-9e8b-3ab262ef1fb7" containerID="6404f8bf1af89d10c30c8c85d005419ef6e72c50ef56a42c9636d841c9dd3813" exitCode=0
Sep 14 10:18:34.030385 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:34.029988    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl" event={"ID":"3d80c24f-867f-4c36-9e8b-3ab262ef1fb7","Type":"ContainerDied","Data":"6404f8bf1af89d10c30c8c85d005419ef6e72c50ef56a42c9636d841c9dd3813"}
Sep 14 10:18:34.030385 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:34.030077    2563 scope.go:117] "RemoveContainer" containerID="cf1ead24add79491bffa8657c0af3e606ddd578d947b41d76939f11b5f9ff21c"
Sep 14 10:18:34.111969 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:34.111929    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl"
Sep 14 10:18:34.170230 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:34.170130    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d80c24f-867f-4c36-9e8b-3ab262ef1fb7-istiod-ca-cert\") pod \"3d80c24f-867f-4c36-9e8b-3ab262ef1fb7\" (UID: \"3d80c24f-867f-4c36-9e8b-3ab262ef1fb7\") "
Sep 14 10:18:34.170230 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:34.170175    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d80c24f-867f-4c36-9e8b-3ab262ef1fb7-istio-data\") pod \"3d80c24f-867f-4c36-9e8b-3ab262ef1fb7\" (UID: \"3d80c24f-867f-4c36-9e8b-3ab262ef1fb7\") "
Sep 14 10:18:34.170230 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:34.170199    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-29szt\" (UniqueName: \"kubernetes.io/projected/3d80c24f-867f-4c36-9e8b-3ab262ef1fb7-kube-api-access-29szt\") pod \"3d80c24f-867f-4c36-9e8b-3ab262ef1fb7\" (UID: \"3d80c24f-867f-4c36-9e8b-3ab262ef1fb7\") "
Sep 14 10:18:34.170508 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:34.170400    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d80c24f-867f-4c36-9e8b-3ab262ef1fb7-credential-socket\") pod \"3d80c24f-867f-4c36-9e8b-3ab262ef1fb7\" (UID: \"3d80c24f-867f-4c36-9e8b-3ab262ef1fb7\") "
Sep 14 10:18:34.170508 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:34.170451    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d80c24f-867f-4c36-9e8b-3ab262ef1fb7-istio-data" (OuterVolumeSpecName: "istio-data") pod "3d80c24f-867f-4c36-9e8b-3ab262ef1fb7" (UID: "3d80c24f-867f-4c36-9e8b-3ab262ef1fb7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:18:34.170508 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:34.170462    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d80c24f-867f-4c36-9e8b-3ab262ef1fb7-istio-podinfo\") pod \"3d80c24f-867f-4c36-9e8b-3ab262ef1fb7\" (UID: \"3d80c24f-867f-4c36-9e8b-3ab262ef1fb7\") "
Sep 14 10:18:34.170673 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:34.170514    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d80c24f-867f-4c36-9e8b-3ab262ef1fb7-workload-certs\") pod \"3d80c24f-867f-4c36-9e8b-3ab262ef1fb7\" (UID: \"3d80c24f-867f-4c36-9e8b-3ab262ef1fb7\") "
Sep 14 10:18:34.170673 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:34.170548    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d80c24f-867f-4c36-9e8b-3ab262ef1fb7-istio-envoy\") pod \"3d80c24f-867f-4c36-9e8b-3ab262ef1fb7\" (UID: \"3d80c24f-867f-4c36-9e8b-3ab262ef1fb7\") "
Sep 14 10:18:34.170673 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:34.170568    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d80c24f-867f-4c36-9e8b-3ab262ef1fb7-workload-socket\") pod \"3d80c24f-867f-4c36-9e8b-3ab262ef1fb7\" (UID: \"3d80c24f-867f-4c36-9e8b-3ab262ef1fb7\") "
Sep 14 10:18:34.170673 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:34.170625    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d80c24f-867f-4c36-9e8b-3ab262ef1fb7-istio-token\") pod \"3d80c24f-867f-4c36-9e8b-3ab262ef1fb7\" (UID: \"3d80c24f-867f-4c36-9e8b-3ab262ef1fb7\") "
Sep 14 10:18:34.170673 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:34.170655    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d80c24f-867f-4c36-9e8b-3ab262ef1fb7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3d80c24f-867f-4c36-9e8b-3ab262ef1fb7" (UID: "3d80c24f-867f-4c36-9e8b-3ab262ef1fb7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:18:34.170946 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:34.170677    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d80c24f-867f-4c36-9e8b-3ab262ef1fb7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3d80c24f-867f-4c36-9e8b-3ab262ef1fb7" (UID: "3d80c24f-867f-4c36-9e8b-3ab262ef1fb7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:18:34.170946 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:34.170811    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d80c24f-867f-4c36-9e8b-3ab262ef1fb7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3d80c24f-867f-4c36-9e8b-3ab262ef1fb7" (UID: "3d80c24f-867f-4c36-9e8b-3ab262ef1fb7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:18:34.170946 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:34.170874    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d80c24f-867f-4c36-9e8b-3ab262ef1fb7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3d80c24f-867f-4c36-9e8b-3ab262ef1fb7" (UID: "3d80c24f-867f-4c36-9e8b-3ab262ef1fb7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:18:34.171157 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:34.170977    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d80c24f-867f-4c36-9e8b-3ab262ef1fb7-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:18:34.171157 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:34.170996    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d80c24f-867f-4c36-9e8b-3ab262ef1fb7-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:18:34.171157 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:34.171036    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d80c24f-867f-4c36-9e8b-3ab262ef1fb7-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:18:34.171157 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:34.171044    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d80c24f-867f-4c36-9e8b-3ab262ef1fb7-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:18:34.171157 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:34.171052    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d80c24f-867f-4c36-9e8b-3ab262ef1fb7-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:18:34.172521 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:34.172488    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d80c24f-867f-4c36-9e8b-3ab262ef1fb7-kube-api-access-29szt" (OuterVolumeSpecName: "kube-api-access-29szt") pod "3d80c24f-867f-4c36-9e8b-3ab262ef1fb7" (UID: "3d80c24f-867f-4c36-9e8b-3ab262ef1fb7"). InnerVolumeSpecName "kube-api-access-29szt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:18:34.172639 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:34.172592    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d80c24f-867f-4c36-9e8b-3ab262ef1fb7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3d80c24f-867f-4c36-9e8b-3ab262ef1fb7" (UID: "3d80c24f-867f-4c36-9e8b-3ab262ef1fb7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 10:18:34.173092 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:34.173068    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d80c24f-867f-4c36-9e8b-3ab262ef1fb7-istio-token" (OuterVolumeSpecName: "istio-token") pod "3d80c24f-867f-4c36-9e8b-3ab262ef1fb7" (UID: "3d80c24f-867f-4c36-9e8b-3ab262ef1fb7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:18:34.173206 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:34.173105    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d80c24f-867f-4c36-9e8b-3ab262ef1fb7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3d80c24f-867f-4c36-9e8b-3ab262ef1fb7" (UID: "3d80c24f-867f-4c36-9e8b-3ab262ef1fb7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:18:34.271994 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:34.271950    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d80c24f-867f-4c36-9e8b-3ab262ef1fb7-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:18:34.271994 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:34.271989    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29szt\" (UniqueName: \"kubernetes.io/projected/3d80c24f-867f-4c36-9e8b-3ab262ef1fb7-kube-api-access-29szt\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:18:34.272229 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:34.272033    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d80c24f-867f-4c36-9e8b-3ab262ef1fb7-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:18:34.272229 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:34.272044    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d80c24f-867f-4c36-9e8b-3ab262ef1fb7-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:18:35.037383 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:35.037348    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl"
Sep 14 10:18:35.037869 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:35.037382    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl" event={"ID":"3d80c24f-867f-4c36-9e8b-3ab262ef1fb7","Type":"ContainerDied","Data":"c9675e4b528abddf1c8692f46b8525688d555be4466eab5594b18b7cd6b43e2d"}
Sep 14 10:18:35.037869 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:35.037456    2563 scope.go:117] "RemoveContainer" containerID="6404f8bf1af89d10c30c8c85d005419ef6e72c50ef56a42c9636d841c9dd3813"
Sep 14 10:18:35.060332 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:35.060287    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl"]
Sep 14 10:18:35.066561 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:35.066520    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f4d054-openshift-default-5569975444-v5xpl"]
Sep 14 10:18:35.482442 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:18:35.482342    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d80c24f-867f-4c36-9e8b-3ab262ef1fb7" path="/var/lib/kubelet/pods/3d80c24f-867f-4c36-9e8b-3ab262ef1fb7/volumes"
Sep 14 10:19:56.352771 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:19:56.352735    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:19:57.861618 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:19:57.861582    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:20:17.547276 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:20:17.547232    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:20:17.647842 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:20:17.647763    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:20:55.998764 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:20:55.998727    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:20:56.801451 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:20:56.801413    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:21:32.242273 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:21:32.242233    2563 scope.go:117] "RemoveContainer" containerID="728ecf78453282da53262fe623d23090745bdf57108e49e288dd06946fc7674e"
Sep 14 10:21:32.253902 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:21:32.253856    2563 scope.go:117] "RemoveContainer" containerID="c15a4c144ca1056199a3d0633999d68d18e829e8516ca02e37c00a73a0e63543"
Sep 14 10:21:32.263166 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:21:32.263138    2563 scope.go:117] "RemoveContainer" containerID="9fb0bed13a80a6119dcf8ec136f5aa8f249193f8f4164ae0c980eda9b5036632"
Sep 14 10:22:32.294582 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:22:32.294549    2563 scope.go:117] "RemoveContainer" containerID="262a09290e83d6483616f9137ec61e2f457a0c2f1f4bc1554a6103ead3a2558d"
Sep 14 10:22:32.302943 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:22:32.302922    2563 scope.go:117] "RemoveContainer" containerID="5eb5696cd408662fdc6cd7d3a4c6fcd34f2471009c42a85b689086a48a8d3d59"
Sep 14 10:22:43.064825 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:22:43.064788    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-82942479-kserve-86bfcfb9d7-cd2ct"]
Sep 14 10:22:43.065204 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:22:43.065132    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1bb4690-068b-43c3-abdb-8bcda1591ae4" containerName="istio-proxy"
Sep 14 10:22:43.065204 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:22:43.065146    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1bb4690-068b-43c3-abdb-8bcda1591ae4" containerName="istio-proxy"
Sep 14 10:22:43.065204 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:22:43.065162    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51e15aea-5153-46bc-b34c-e186355d15f6" containerName="istio-proxy"
Sep 14 10:22:43.065204 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:22:43.065168    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="51e15aea-5153-46bc-b34c-e186355d15f6" containerName="istio-proxy"
Sep 14 10:22:43.065204 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:22:43.065176    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d80c24f-867f-4c36-9e8b-3ab262ef1fb7" containerName="istio-proxy"
Sep 14 10:22:43.065204 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:22:43.065181    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d80c24f-867f-4c36-9e8b-3ab262ef1fb7" containerName="istio-proxy"
Sep 14 10:22:43.065204 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:22:43.065192    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d80c24f-867f-4c36-9e8b-3ab262ef1fb7" containerName="istio-proxy"
Sep 14 10:22:43.065204 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:22:43.065197    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d80c24f-867f-4c36-9e8b-3ab262ef1fb7" containerName="istio-proxy"
Sep 14 10:22:43.065204 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:22:43.065205    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2784d964-c759-4d84-8fd8-99ad6664c4da" containerName="revoke-keys"
Sep 14 10:22:43.065204 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:22:43.065210    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2784d964-c759-4d84-8fd8-99ad6664c4da" containerName="revoke-keys"
Sep 14 10:22:43.065529 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:22:43.065262    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2784d964-c759-4d84-8fd8-99ad6664c4da" containerName="revoke-keys"
Sep 14 10:22:43.065529 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:22:43.065271    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1bb4690-068b-43c3-abdb-8bcda1591ae4" containerName="istio-proxy"
Sep 14 10:22:43.065529 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:22:43.065277    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1bb4690-068b-43c3-abdb-8bcda1591ae4" containerName="istio-proxy"
Sep 14 10:22:43.065529 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:22:43.065283    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d80c24f-867f-4c36-9e8b-3ab262ef1fb7" containerName="istio-proxy"
Sep 14 10:22:43.065529 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:22:43.065289    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="51e15aea-5153-46bc-b34c-e186355d15f6" containerName="istio-proxy"
Sep 14 10:22:43.065529 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:22:43.065295    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d80c24f-867f-4c36-9e8b-3ab262ef1fb7" containerName="istio-proxy"
Sep 14 10:22:43.065529 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:22:43.065300    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="51e15aea-5153-46bc-b34c-e186355d15f6" containerName="istio-proxy"
Sep 14 10:22:43.068785 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:22:43.068758    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-82942479-kserve-86bfcfb9d7-cd2ct"
Sep 14 10:22:43.070831 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:22:43.070807    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-82942479-kserve-self-signed-certs\""
Sep 14 10:22:43.077507 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:22:43.077480    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-82942479-kserve-86bfcfb9d7-cd2ct"]
Sep 14 10:22:43.113615 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:22:43.113582    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:22:43.206019 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:22:43.205969    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c11f6996-1c88-499e-9903-7b230387a49f-model-cache\") pod \"e2e-same-id-a-82942479-kserve-86bfcfb9d7-cd2ct\" (UID: \"c11f6996-1c88-499e-9903-7b230387a49f\") " pod="llm/e2e-same-id-a-82942479-kserve-86bfcfb9d7-cd2ct"
Sep 14 10:22:43.206184 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:22:43.206034    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c11f6996-1c88-499e-9903-7b230387a49f-dshm\") pod \"e2e-same-id-a-82942479-kserve-86bfcfb9d7-cd2ct\" (UID: \"c11f6996-1c88-499e-9903-7b230387a49f\") " pod="llm/e2e-same-id-a-82942479-kserve-86bfcfb9d7-cd2ct"
Sep 14 10:22:43.206184 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:22:43.206081    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c11f6996-1c88-499e-9903-7b230387a49f-home\") pod \"e2e-same-id-a-82942479-kserve-86bfcfb9d7-cd2ct\" (UID: \"c11f6996-1c88-499e-9903-7b230387a49f\") " pod="llm/e2e-same-id-a-82942479-kserve-86bfcfb9d7-cd2ct"
Sep 14 10:22:43.206184 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:22:43.206100    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xhgs\" (UniqueName: \"kubernetes.io/projected/c11f6996-1c88-499e-9903-7b230387a49f-kube-api-access-7xhgs\") pod \"e2e-same-id-a-82942479-kserve-86bfcfb9d7-cd2ct\" (UID: \"c11f6996-1c88-499e-9903-7b230387a49f\") " pod="llm/e2e-same-id-a-82942479-kserve-86bfcfb9d7-cd2ct"
Sep 14 10:22:43.206360 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:22:43.206234    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c11f6996-1c88-499e-9903-7b230387a49f-tmp-dir\") pod \"e2e-same-id-a-82942479-kserve-86bfcfb9d7-cd2ct\" (UID: \"c11f6996-1c88-499e-9903-7b230387a49f\") " pod="llm/e2e-same-id-a-82942479-kserve-86bfcfb9d7-cd2ct"
Sep 14 10:22:43.206360 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:22:43.206295    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c11f6996-1c88-499e-9903-7b230387a49f-tls-certs\") pod \"e2e-same-id-a-82942479-kserve-86bfcfb9d7-cd2ct\" (UID: \"c11f6996-1c88-499e-9903-7b230387a49f\") " pod="llm/e2e-same-id-a-82942479-kserve-86bfcfb9d7-cd2ct"
Sep 14 10:22:43.307094 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:22:43.307058    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c11f6996-1c88-499e-9903-7b230387a49f-model-cache\") pod \"e2e-same-id-a-82942479-kserve-86bfcfb9d7-cd2ct\" (UID: \"c11f6996-1c88-499e-9903-7b230387a49f\") " pod="llm/e2e-same-id-a-82942479-kserve-86bfcfb9d7-cd2ct"
Sep 14 10:22:43.307094 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:22:43.307099    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c11f6996-1c88-499e-9903-7b230387a49f-dshm\") pod \"e2e-same-id-a-82942479-kserve-86bfcfb9d7-cd2ct\" (UID: \"c11f6996-1c88-499e-9903-7b230387a49f\") " pod="llm/e2e-same-id-a-82942479-kserve-86bfcfb9d7-cd2ct"
Sep 14 10:22:43.307348 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:22:43.307138    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c11f6996-1c88-499e-9903-7b230387a49f-home\") pod \"e2e-same-id-a-82942479-kserve-86bfcfb9d7-cd2ct\" (UID: \"c11f6996-1c88-499e-9903-7b230387a49f\") " pod="llm/e2e-same-id-a-82942479-kserve-86bfcfb9d7-cd2ct"
Sep 14 10:22:43.307348 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:22:43.307154    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xhgs\" (UniqueName: \"kubernetes.io/projected/c11f6996-1c88-499e-9903-7b230387a49f-kube-api-access-7xhgs\") pod \"e2e-same-id-a-82942479-kserve-86bfcfb9d7-cd2ct\" (UID: \"c11f6996-1c88-499e-9903-7b230387a49f\") " pod="llm/e2e-same-id-a-82942479-kserve-86bfcfb9d7-cd2ct"
Sep 14 10:22:43.307348 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:22:43.307208    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c11f6996-1c88-499e-9903-7b230387a49f-tmp-dir\") pod \"e2e-same-id-a-82942479-kserve-86bfcfb9d7-cd2ct\" (UID: \"c11f6996-1c88-499e-9903-7b230387a49f\") " pod="llm/e2e-same-id-a-82942479-kserve-86bfcfb9d7-cd2ct"
Sep 14 10:22:43.307348 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:22:43.307250    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c11f6996-1c88-499e-9903-7b230387a49f-tls-certs\") pod \"e2e-same-id-a-82942479-kserve-86bfcfb9d7-cd2ct\" (UID: \"c11f6996-1c88-499e-9903-7b230387a49f\") " pod="llm/e2e-same-id-a-82942479-kserve-86bfcfb9d7-cd2ct"
Sep 14 10:22:43.307570 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:22:43.307515    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c11f6996-1c88-499e-9903-7b230387a49f-model-cache\") pod \"e2e-same-id-a-82942479-kserve-86bfcfb9d7-cd2ct\" (UID: \"c11f6996-1c88-499e-9903-7b230387a49f\") " pod="llm/e2e-same-id-a-82942479-kserve-86bfcfb9d7-cd2ct"
Sep 14 10:22:43.307617 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:22:43.307589    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c11f6996-1c88-499e-9903-7b230387a49f-home\") pod \"e2e-same-id-a-82942479-kserve-86bfcfb9d7-cd2ct\" (UID: \"c11f6996-1c88-499e-9903-7b230387a49f\") " pod="llm/e2e-same-id-a-82942479-kserve-86bfcfb9d7-cd2ct"
Sep 14 10:22:43.307689 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:22:43.307632    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c11f6996-1c88-499e-9903-7b230387a49f-tmp-dir\") pod \"e2e-same-id-a-82942479-kserve-86bfcfb9d7-cd2ct\" (UID: \"c11f6996-1c88-499e-9903-7b230387a49f\") " pod="llm/e2e-same-id-a-82942479-kserve-86bfcfb9d7-cd2ct"
Sep 14 10:22:43.309451 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:22:43.309428    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c11f6996-1c88-499e-9903-7b230387a49f-dshm\") pod \"e2e-same-id-a-82942479-kserve-86bfcfb9d7-cd2ct\" (UID: \"c11f6996-1c88-499e-9903-7b230387a49f\") " pod="llm/e2e-same-id-a-82942479-kserve-86bfcfb9d7-cd2ct"
Sep 14 10:22:43.309698 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:22:43.309680    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c11f6996-1c88-499e-9903-7b230387a49f-tls-certs\") pod \"e2e-same-id-a-82942479-kserve-86bfcfb9d7-cd2ct\" (UID: \"c11f6996-1c88-499e-9903-7b230387a49f\") " pod="llm/e2e-same-id-a-82942479-kserve-86bfcfb9d7-cd2ct"
Sep 14 10:22:43.315403 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:22:43.315349    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xhgs\" (UniqueName: \"kubernetes.io/projected/c11f6996-1c88-499e-9903-7b230387a49f-kube-api-access-7xhgs\") pod \"e2e-same-id-a-82942479-kserve-86bfcfb9d7-cd2ct\" (UID: \"c11f6996-1c88-499e-9903-7b230387a49f\") " pod="llm/e2e-same-id-a-82942479-kserve-86bfcfb9d7-cd2ct"
Sep 14 10:22:43.380147 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:22:43.380117    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-82942479-kserve-86bfcfb9d7-cd2ct"
Sep 14 10:22:43.651924 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:22:43.651896    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-82942479-kserve-86bfcfb9d7-cd2ct"]
Sep 14 10:22:43.653544 ip-10-0-132-4 kubenswrapper[2563]: W0914 10:22:43.653516    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc11f6996_1c88_499e_9903_7b230387a49f.slice/crio-a683154f8af81d69dc44c1b47852fc1483d5b9cda781bcd8e5eef9e2c9dc2042 WatchSource:0}: Error finding container a683154f8af81d69dc44c1b47852fc1483d5b9cda781bcd8e5eef9e2c9dc2042: Status 404 returned error can't find the container with id a683154f8af81d69dc44c1b47852fc1483d5b9cda781bcd8e5eef9e2c9dc2042
Sep 14 10:22:44.462729 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:22:44.462694    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-82942479-kserve-86bfcfb9d7-cd2ct" event={"ID":"c11f6996-1c88-499e-9903-7b230387a49f","Type":"ContainerStarted","Data":"c0234245bdbee1ba7464013a5f01eee5716b10309b28da715ed6a07a51305b4c"}
Sep 14 10:22:44.463182 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:22:44.462744    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-82942479-kserve-86bfcfb9d7-cd2ct" event={"ID":"c11f6996-1c88-499e-9903-7b230387a49f","Type":"ContainerStarted","Data":"a683154f8af81d69dc44c1b47852fc1483d5b9cda781bcd8e5eef9e2c9dc2042"}
Sep 14 10:22:44.480138 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:22:44.480083    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-82942479-kserve-86bfcfb9d7-cd2ct" podStartSLOduration=1.480067563 podStartE2EDuration="1.480067563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:22:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:22:44.477844324 +0000 UTC m=+2353.517518945" watchObservedRunningTime="2026-09-14 10:22:44.480067563 +0000 UTC m=+2353.519742184"
Sep 14 10:22:53.380265 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:22:53.380196    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-82942479-kserve-86bfcfb9d7-cd2ct"
Sep 14 10:22:53.380680 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:22:53.380282    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-82942479-kserve-86bfcfb9d7-cd2ct"
Sep 14 10:22:53.393362 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:22:53.393323    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-82942479-kserve-86bfcfb9d7-cd2ct"
Sep 14 10:22:53.521624 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:22:53.521585    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-82942479-kserve-86bfcfb9d7-cd2ct"
Sep 14 10:22:54.216791 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:22:54.216752    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:23:05.052961 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:05.052925    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-82942479-kserve-85585595c5-kpvwq"]
Sep 14 10:23:05.053566 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:05.053277    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1bb4690-068b-43c3-abdb-8bcda1591ae4" containerName="istio-proxy"
Sep 14 10:23:05.053566 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:05.053290    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1bb4690-068b-43c3-abdb-8bcda1591ae4" containerName="istio-proxy"
Sep 14 10:23:05.053566 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:05.053308    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51e15aea-5153-46bc-b34c-e186355d15f6" containerName="istio-proxy"
Sep 14 10:23:05.053566 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:05.053313    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="51e15aea-5153-46bc-b34c-e186355d15f6" containerName="istio-proxy"
Sep 14 10:23:05.056557 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:05.056535    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-82942479-kserve-85585595c5-kpvwq"
Sep 14 10:23:05.058921 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:05.058890    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-82942479-kserve-self-signed-certs\""
Sep 14 10:23:05.069413 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:05.069387    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-82942479-kserve-85585595c5-kpvwq"]
Sep 14 10:23:05.100801 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:05.100765    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:23:05.106831 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:05.106812    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94cfd911-a61e-4d49-b1c0-e27f54e609e5-tls-certs\") pod \"e2e-same-id-b-82942479-kserve-85585595c5-kpvwq\" (UID: \"94cfd911-a61e-4d49-b1c0-e27f54e609e5\") " pod="llm/e2e-same-id-b-82942479-kserve-85585595c5-kpvwq"
Sep 14 10:23:05.106911 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:05.106849    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94cfd911-a61e-4d49-b1c0-e27f54e609e5-tmp-dir\") pod \"e2e-same-id-b-82942479-kserve-85585595c5-kpvwq\" (UID: \"94cfd911-a61e-4d49-b1c0-e27f54e609e5\") " pod="llm/e2e-same-id-b-82942479-kserve-85585595c5-kpvwq"
Sep 14 10:23:05.106977 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:05.106960    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94cfd911-a61e-4d49-b1c0-e27f54e609e5-model-cache\") pod \"e2e-same-id-b-82942479-kserve-85585595c5-kpvwq\" (UID: \"94cfd911-a61e-4d49-b1c0-e27f54e609e5\") " pod="llm/e2e-same-id-b-82942479-kserve-85585595c5-kpvwq"
Sep 14 10:23:05.107027 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:05.106991    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94cfd911-a61e-4d49-b1c0-e27f54e609e5-home\") pod \"e2e-same-id-b-82942479-kserve-85585595c5-kpvwq\" (UID: \"94cfd911-a61e-4d49-b1c0-e27f54e609e5\") " pod="llm/e2e-same-id-b-82942479-kserve-85585595c5-kpvwq"
Sep 14 10:23:05.107065 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:05.107046    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94cfd911-a61e-4d49-b1c0-e27f54e609e5-dshm\") pod \"e2e-same-id-b-82942479-kserve-85585595c5-kpvwq\" (UID: \"94cfd911-a61e-4d49-b1c0-e27f54e609e5\") " pod="llm/e2e-same-id-b-82942479-kserve-85585595c5-kpvwq"
Sep 14 10:23:05.107100 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:05.107068    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bhwb\" (UniqueName: \"kubernetes.io/projected/94cfd911-a61e-4d49-b1c0-e27f54e609e5-kube-api-access-5bhwb\") pod \"e2e-same-id-b-82942479-kserve-85585595c5-kpvwq\" (UID: \"94cfd911-a61e-4d49-b1c0-e27f54e609e5\") " pod="llm/e2e-same-id-b-82942479-kserve-85585595c5-kpvwq"
Sep 14 10:23:05.208109 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:05.208067    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94cfd911-a61e-4d49-b1c0-e27f54e609e5-model-cache\") pod \"e2e-same-id-b-82942479-kserve-85585595c5-kpvwq\" (UID: \"94cfd911-a61e-4d49-b1c0-e27f54e609e5\") " pod="llm/e2e-same-id-b-82942479-kserve-85585595c5-kpvwq"
Sep 14 10:23:05.208290 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:05.208112    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94cfd911-a61e-4d49-b1c0-e27f54e609e5-home\") pod \"e2e-same-id-b-82942479-kserve-85585595c5-kpvwq\" (UID: \"94cfd911-a61e-4d49-b1c0-e27f54e609e5\") " pod="llm/e2e-same-id-b-82942479-kserve-85585595c5-kpvwq"
Sep 14 10:23:05.208290 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:05.208151    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94cfd911-a61e-4d49-b1c0-e27f54e609e5-dshm\") pod \"e2e-same-id-b-82942479-kserve-85585595c5-kpvwq\" (UID: \"94cfd911-a61e-4d49-b1c0-e27f54e609e5\") " pod="llm/e2e-same-id-b-82942479-kserve-85585595c5-kpvwq"
Sep 14 10:23:05.208290 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:05.208173    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bhwb\" (UniqueName: \"kubernetes.io/projected/94cfd911-a61e-4d49-b1c0-e27f54e609e5-kube-api-access-5bhwb\") pod \"e2e-same-id-b-82942479-kserve-85585595c5-kpvwq\" (UID: \"94cfd911-a61e-4d49-b1c0-e27f54e609e5\") " pod="llm/e2e-same-id-b-82942479-kserve-85585595c5-kpvwq"
Sep 14 10:23:05.208290 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:05.208233    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94cfd911-a61e-4d49-b1c0-e27f54e609e5-tls-certs\") pod \"e2e-same-id-b-82942479-kserve-85585595c5-kpvwq\" (UID: \"94cfd911-a61e-4d49-b1c0-e27f54e609e5\") " pod="llm/e2e-same-id-b-82942479-kserve-85585595c5-kpvwq"
Sep 14 10:23:05.208290 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:05.208269    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94cfd911-a61e-4d49-b1c0-e27f54e609e5-tmp-dir\") pod \"e2e-same-id-b-82942479-kserve-85585595c5-kpvwq\" (UID: \"94cfd911-a61e-4d49-b1c0-e27f54e609e5\") " pod="llm/e2e-same-id-b-82942479-kserve-85585595c5-kpvwq"
Sep 14 10:23:05.208556 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:05.208531    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94cfd911-a61e-4d49-b1c0-e27f54e609e5-model-cache\") pod \"e2e-same-id-b-82942479-kserve-85585595c5-kpvwq\" (UID: \"94cfd911-a61e-4d49-b1c0-e27f54e609e5\") " pod="llm/e2e-same-id-b-82942479-kserve-85585595c5-kpvwq"
Sep 14 10:23:05.208622 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:05.208556    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94cfd911-a61e-4d49-b1c0-e27f54e609e5-home\") pod \"e2e-same-id-b-82942479-kserve-85585595c5-kpvwq\" (UID: \"94cfd911-a61e-4d49-b1c0-e27f54e609e5\") " pod="llm/e2e-same-id-b-82942479-kserve-85585595c5-kpvwq"
Sep 14 10:23:05.208675 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:05.208632    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94cfd911-a61e-4d49-b1c0-e27f54e609e5-tmp-dir\") pod \"e2e-same-id-b-82942479-kserve-85585595c5-kpvwq\" (UID: \"94cfd911-a61e-4d49-b1c0-e27f54e609e5\") " pod="llm/e2e-same-id-b-82942479-kserve-85585595c5-kpvwq"
Sep 14 10:23:05.210523 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:05.210498    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94cfd911-a61e-4d49-b1c0-e27f54e609e5-dshm\") pod \"e2e-same-id-b-82942479-kserve-85585595c5-kpvwq\" (UID: \"94cfd911-a61e-4d49-b1c0-e27f54e609e5\") " pod="llm/e2e-same-id-b-82942479-kserve-85585595c5-kpvwq"
Sep 14 10:23:05.210710 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:05.210629    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94cfd911-a61e-4d49-b1c0-e27f54e609e5-tls-certs\") pod \"e2e-same-id-b-82942479-kserve-85585595c5-kpvwq\" (UID: \"94cfd911-a61e-4d49-b1c0-e27f54e609e5\") " pod="llm/e2e-same-id-b-82942479-kserve-85585595c5-kpvwq"
Sep 14 10:23:05.215200 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:05.215177    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bhwb\" (UniqueName: \"kubernetes.io/projected/94cfd911-a61e-4d49-b1c0-e27f54e609e5-kube-api-access-5bhwb\") pod \"e2e-same-id-b-82942479-kserve-85585595c5-kpvwq\" (UID: \"94cfd911-a61e-4d49-b1c0-e27f54e609e5\") " pod="llm/e2e-same-id-b-82942479-kserve-85585595c5-kpvwq"
Sep 14 10:23:05.366630 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:05.366543    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-82942479-kserve-85585595c5-kpvwq"
Sep 14 10:23:05.640672 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:05.640644    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-82942479-kserve-85585595c5-kpvwq"]
Sep 14 10:23:05.642168 ip-10-0-132-4 kubenswrapper[2563]: W0914 10:23:05.642129    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod94cfd911_a61e_4d49_b1c0_e27f54e609e5.slice/crio-d84110ddbe6c31bddbc8fe30ff7923c0ea203b107ba13a151201a4347a3a6b3a WatchSource:0}: Error finding container d84110ddbe6c31bddbc8fe30ff7923c0ea203b107ba13a151201a4347a3a6b3a: Status 404 returned error can't find the container with id d84110ddbe6c31bddbc8fe30ff7923c0ea203b107ba13a151201a4347a3a6b3a
Sep 14 10:23:06.585736 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:06.585697    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-82942479-kserve-85585595c5-kpvwq" event={"ID":"94cfd911-a61e-4d49-b1c0-e27f54e609e5","Type":"ContainerStarted","Data":"df643f9ce62433be3e7ca60117421bce05037c3dceabf1e9da073f62be987226"}
Sep 14 10:23:06.585736 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:06.585745    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-82942479-kserve-85585595c5-kpvwq" event={"ID":"94cfd911-a61e-4d49-b1c0-e27f54e609e5","Type":"ContainerStarted","Data":"d84110ddbe6c31bddbc8fe30ff7923c0ea203b107ba13a151201a4347a3a6b3a"}
Sep 14 10:23:06.601614 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:06.601561    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-82942479-kserve-85585595c5-kpvwq" podStartSLOduration=1.601542759 podStartE2EDuration="1.601542759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:23:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:23:06.600989443 +0000 UTC m=+2375.640664064" watchObservedRunningTime="2026-09-14 10:23:06.601542759 +0000 UTC m=+2375.641217381"
Sep 14 10:23:15.367489 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:15.367451    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-82942479-kserve-85585595c5-kpvwq"
Sep 14 10:23:15.368137 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:15.367514    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-82942479-kserve-85585595c5-kpvwq"
Sep 14 10:23:15.380130 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:15.380102    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-82942479-kserve-85585595c5-kpvwq"
Sep 14 10:23:15.644458 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:15.644379    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-82942479-kserve-85585595c5-kpvwq"
Sep 14 10:23:27.076133 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:27.076084    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-82942479-kserve-86bfcfb9d7-cd2ct"]
Sep 14 10:23:27.095940 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:27.095906    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:23:28.746603 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:28.746566    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-82942479-kserve-85585595c5-kpvwq"]
Sep 14 10:23:28.765565 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:28.765531    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:23:42.095622 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:42.095577    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-82942479-kserve-86bfcfb9d7-cd2ct" podUID="c11f6996-1c88-499e-9903-7b230387a49f" containerName="main" containerID="cri-o://c0234245bdbee1ba7464013a5f01eee5716b10309b28da715ed6a07a51305b4c" gracePeriod=45
Sep 14 10:23:42.347033 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:42.346961    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-82942479-kserve-86bfcfb9d7-cd2ct"
Sep 14 10:23:42.433294 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:42.433252    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7xhgs\" (UniqueName: \"kubernetes.io/projected/c11f6996-1c88-499e-9903-7b230387a49f-kube-api-access-7xhgs\") pod \"c11f6996-1c88-499e-9903-7b230387a49f\" (UID: \"c11f6996-1c88-499e-9903-7b230387a49f\") "
Sep 14 10:23:42.433294 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:42.433300    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c11f6996-1c88-499e-9903-7b230387a49f-model-cache\") pod \"c11f6996-1c88-499e-9903-7b230387a49f\" (UID: \"c11f6996-1c88-499e-9903-7b230387a49f\") "
Sep 14 10:23:42.433561 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:42.433319    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c11f6996-1c88-499e-9903-7b230387a49f-home\") pod \"c11f6996-1c88-499e-9903-7b230387a49f\" (UID: \"c11f6996-1c88-499e-9903-7b230387a49f\") "
Sep 14 10:23:42.433561 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:42.433360    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c11f6996-1c88-499e-9903-7b230387a49f-tmp-dir\") pod \"c11f6996-1c88-499e-9903-7b230387a49f\" (UID: \"c11f6996-1c88-499e-9903-7b230387a49f\") "
Sep 14 10:23:42.433561 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:42.433423    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c11f6996-1c88-499e-9903-7b230387a49f-dshm\") pod \"c11f6996-1c88-499e-9903-7b230387a49f\" (UID: \"c11f6996-1c88-499e-9903-7b230387a49f\") "
Sep 14 10:23:42.433561 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:42.433442    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c11f6996-1c88-499e-9903-7b230387a49f-tls-certs\") pod \"c11f6996-1c88-499e-9903-7b230387a49f\" (UID: \"c11f6996-1c88-499e-9903-7b230387a49f\") "
Sep 14 10:23:42.433773 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:42.433630    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c11f6996-1c88-499e-9903-7b230387a49f-model-cache" (OuterVolumeSpecName: "model-cache") pod "c11f6996-1c88-499e-9903-7b230387a49f" (UID: "c11f6996-1c88-499e-9903-7b230387a49f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:23:42.433773 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:42.433656    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c11f6996-1c88-499e-9903-7b230387a49f-home" (OuterVolumeSpecName: "home") pod "c11f6996-1c88-499e-9903-7b230387a49f" (UID: "c11f6996-1c88-499e-9903-7b230387a49f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:23:42.433773 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:42.433686    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c11f6996-1c88-499e-9903-7b230387a49f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c11f6996-1c88-499e-9903-7b230387a49f" (UID: "c11f6996-1c88-499e-9903-7b230387a49f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:23:42.435562 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:42.435534    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c11f6996-1c88-499e-9903-7b230387a49f-dshm" (OuterVolumeSpecName: "dshm") pod "c11f6996-1c88-499e-9903-7b230387a49f" (UID: "c11f6996-1c88-499e-9903-7b230387a49f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:23:42.435686 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:42.435571    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c11f6996-1c88-499e-9903-7b230387a49f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c11f6996-1c88-499e-9903-7b230387a49f" (UID: "c11f6996-1c88-499e-9903-7b230387a49f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:23:42.435686 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:42.435669    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c11f6996-1c88-499e-9903-7b230387a49f-kube-api-access-7xhgs" (OuterVolumeSpecName: "kube-api-access-7xhgs") pod "c11f6996-1c88-499e-9903-7b230387a49f" (UID: "c11f6996-1c88-499e-9903-7b230387a49f"). InnerVolumeSpecName "kube-api-access-7xhgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:23:42.534379 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:42.534316    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c11f6996-1c88-499e-9903-7b230387a49f-dshm\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:23:42.534379 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:42.534367    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c11f6996-1c88-499e-9903-7b230387a49f-tls-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:23:42.534379 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:42.534383    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xhgs\" (UniqueName: \"kubernetes.io/projected/c11f6996-1c88-499e-9903-7b230387a49f-kube-api-access-7xhgs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:23:42.534379 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:42.534397    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c11f6996-1c88-499e-9903-7b230387a49f-model-cache\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:23:42.534729 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:42.534410    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c11f6996-1c88-499e-9903-7b230387a49f-home\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:23:42.534729 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:42.534422    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c11f6996-1c88-499e-9903-7b230387a49f-tmp-dir\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:23:42.779220 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:42.779187    2563 generic.go:356] "Generic (PLEG): container finished" podID="c11f6996-1c88-499e-9903-7b230387a49f" containerID="c0234245bdbee1ba7464013a5f01eee5716b10309b28da715ed6a07a51305b4c" exitCode=0
Sep 14 10:23:42.779220 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:42.779232    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-82942479-kserve-86bfcfb9d7-cd2ct"
Sep 14 10:23:42.779485 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:42.779250    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-82942479-kserve-86bfcfb9d7-cd2ct" event={"ID":"c11f6996-1c88-499e-9903-7b230387a49f","Type":"ContainerDied","Data":"c0234245bdbee1ba7464013a5f01eee5716b10309b28da715ed6a07a51305b4c"}
Sep 14 10:23:42.779485 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:42.779292    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-82942479-kserve-86bfcfb9d7-cd2ct" event={"ID":"c11f6996-1c88-499e-9903-7b230387a49f","Type":"ContainerDied","Data":"a683154f8af81d69dc44c1b47852fc1483d5b9cda781bcd8e5eef9e2c9dc2042"}
Sep 14 10:23:42.779485 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:42.779314    2563 scope.go:117] "RemoveContainer" containerID="c0234245bdbee1ba7464013a5f01eee5716b10309b28da715ed6a07a51305b4c"
Sep 14 10:23:42.788907 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:42.788885    2563 scope.go:117] "RemoveContainer" containerID="c0234245bdbee1ba7464013a5f01eee5716b10309b28da715ed6a07a51305b4c"
Sep 14 10:23:42.789187 ip-10-0-132-4 kubenswrapper[2563]: E0914 10:23:42.789167    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0234245bdbee1ba7464013a5f01eee5716b10309b28da715ed6a07a51305b4c\": container with ID starting with c0234245bdbee1ba7464013a5f01eee5716b10309b28da715ed6a07a51305b4c not found: ID does not exist" containerID="c0234245bdbee1ba7464013a5f01eee5716b10309b28da715ed6a07a51305b4c"
Sep 14 10:23:42.789260 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:42.789195    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0234245bdbee1ba7464013a5f01eee5716b10309b28da715ed6a07a51305b4c"} err="failed to get container status \"c0234245bdbee1ba7464013a5f01eee5716b10309b28da715ed6a07a51305b4c\": rpc error: code = NotFound desc = could not find container \"c0234245bdbee1ba7464013a5f01eee5716b10309b28da715ed6a07a51305b4c\": container with ID starting with c0234245bdbee1ba7464013a5f01eee5716b10309b28da715ed6a07a51305b4c not found: ID does not exist"
Sep 14 10:23:42.801076 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:42.801054    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-82942479-kserve-86bfcfb9d7-cd2ct"]
Sep 14 10:23:42.808512 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:42.808489    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-82942479-kserve-86bfcfb9d7-cd2ct"]
Sep 14 10:23:43.482466 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:43.482432    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c11f6996-1c88-499e-9903-7b230387a49f" path="/var/lib/kubelet/pods/c11f6996-1c88-499e-9903-7b230387a49f/volumes"
Sep 14 10:23:43.767520 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:43.767475    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-82942479-kserve-85585595c5-kpvwq" podUID="94cfd911-a61e-4d49-b1c0-e27f54e609e5" containerName="main" containerID="cri-o://df643f9ce62433be3e7ca60117421bce05037c3dceabf1e9da073f62be987226" gracePeriod=45
Sep 14 10:23:44.023112 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:44.023054    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-82942479-kserve-85585595c5-kpvwq"
Sep 14 10:23:44.047172 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:44.047144    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94cfd911-a61e-4d49-b1c0-e27f54e609e5-home\") pod \"94cfd911-a61e-4d49-b1c0-e27f54e609e5\" (UID: \"94cfd911-a61e-4d49-b1c0-e27f54e609e5\") "
Sep 14 10:23:44.047335 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:44.047195    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94cfd911-a61e-4d49-b1c0-e27f54e609e5-tmp-dir\") pod \"94cfd911-a61e-4d49-b1c0-e27f54e609e5\" (UID: \"94cfd911-a61e-4d49-b1c0-e27f54e609e5\") "
Sep 14 10:23:44.047335 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:44.047236    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94cfd911-a61e-4d49-b1c0-e27f54e609e5-model-cache\") pod \"94cfd911-a61e-4d49-b1c0-e27f54e609e5\" (UID: \"94cfd911-a61e-4d49-b1c0-e27f54e609e5\") "
Sep 14 10:23:44.047335 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:44.047294    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5bhwb\" (UniqueName: \"kubernetes.io/projected/94cfd911-a61e-4d49-b1c0-e27f54e609e5-kube-api-access-5bhwb\") pod \"94cfd911-a61e-4d49-b1c0-e27f54e609e5\" (UID: \"94cfd911-a61e-4d49-b1c0-e27f54e609e5\") "
Sep 14 10:23:44.047335 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:44.047328    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94cfd911-a61e-4d49-b1c0-e27f54e609e5-dshm\") pod \"94cfd911-a61e-4d49-b1c0-e27f54e609e5\" (UID: \"94cfd911-a61e-4d49-b1c0-e27f54e609e5\") "
Sep 14 10:23:44.047569 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:44.047371    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94cfd911-a61e-4d49-b1c0-e27f54e609e5-tls-certs\") pod \"94cfd911-a61e-4d49-b1c0-e27f54e609e5\" (UID: \"94cfd911-a61e-4d49-b1c0-e27f54e609e5\") "
Sep 14 10:23:44.047569 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:44.047481    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94cfd911-a61e-4d49-b1c0-e27f54e609e5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "94cfd911-a61e-4d49-b1c0-e27f54e609e5" (UID: "94cfd911-a61e-4d49-b1c0-e27f54e609e5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:23:44.047569 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:44.047550    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94cfd911-a61e-4d49-b1c0-e27f54e609e5-model-cache" (OuterVolumeSpecName: "model-cache") pod "94cfd911-a61e-4d49-b1c0-e27f54e609e5" (UID: "94cfd911-a61e-4d49-b1c0-e27f54e609e5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:23:44.047727 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:44.047637    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94cfd911-a61e-4d49-b1c0-e27f54e609e5-tmp-dir\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:23:44.047727 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:44.047655    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94cfd911-a61e-4d49-b1c0-e27f54e609e5-model-cache\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:23:44.047823 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:44.047762    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94cfd911-a61e-4d49-b1c0-e27f54e609e5-home" (OuterVolumeSpecName: "home") pod "94cfd911-a61e-4d49-b1c0-e27f54e609e5" (UID: "94cfd911-a61e-4d49-b1c0-e27f54e609e5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:23:44.049633 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:44.049610    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94cfd911-a61e-4d49-b1c0-e27f54e609e5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "94cfd911-a61e-4d49-b1c0-e27f54e609e5" (UID: "94cfd911-a61e-4d49-b1c0-e27f54e609e5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:23:44.050028 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:44.049983    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94cfd911-a61e-4d49-b1c0-e27f54e609e5-dshm" (OuterVolumeSpecName: "dshm") pod "94cfd911-a61e-4d49-b1c0-e27f54e609e5" (UID: "94cfd911-a61e-4d49-b1c0-e27f54e609e5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:23:44.050130 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:44.050039    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94cfd911-a61e-4d49-b1c0-e27f54e609e5-kube-api-access-5bhwb" (OuterVolumeSpecName: "kube-api-access-5bhwb") pod "94cfd911-a61e-4d49-b1c0-e27f54e609e5" (UID: "94cfd911-a61e-4d49-b1c0-e27f54e609e5"). InnerVolumeSpecName "kube-api-access-5bhwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:23:44.148567 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:44.148512    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bhwb\" (UniqueName: \"kubernetes.io/projected/94cfd911-a61e-4d49-b1c0-e27f54e609e5-kube-api-access-5bhwb\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:23:44.148567 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:44.148560    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94cfd911-a61e-4d49-b1c0-e27f54e609e5-dshm\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:23:44.148567 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:44.148571    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94cfd911-a61e-4d49-b1c0-e27f54e609e5-tls-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:23:44.148567 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:44.148579    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94cfd911-a61e-4d49-b1c0-e27f54e609e5-home\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Sep 14 10:23:44.794952 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:44.794914    2563 generic.go:356] "Generic (PLEG): container finished" podID="94cfd911-a61e-4d49-b1c0-e27f54e609e5" containerID="df643f9ce62433be3e7ca60117421bce05037c3dceabf1e9da073f62be987226" exitCode=0
Sep 14 10:23:44.795453 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:44.794985    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-82942479-kserve-85585595c5-kpvwq" event={"ID":"94cfd911-a61e-4d49-b1c0-e27f54e609e5","Type":"ContainerDied","Data":"df643f9ce62433be3e7ca60117421bce05037c3dceabf1e9da073f62be987226"}
Sep 14 10:23:44.795453 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:44.795034    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-82942479-kserve-85585595c5-kpvwq" event={"ID":"94cfd911-a61e-4d49-b1c0-e27f54e609e5","Type":"ContainerDied","Data":"d84110ddbe6c31bddbc8fe30ff7923c0ea203b107ba13a151201a4347a3a6b3a"}
Sep 14 10:23:44.795453 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:44.795051    2563 scope.go:117] "RemoveContainer" containerID="df643f9ce62433be3e7ca60117421bce05037c3dceabf1e9da073f62be987226"
Sep 14 10:23:44.795453 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:44.794985    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-82942479-kserve-85585595c5-kpvwq"
Sep 14 10:23:44.806809 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:44.806788    2563 scope.go:117] "RemoveContainer" containerID="df643f9ce62433be3e7ca60117421bce05037c3dceabf1e9da073f62be987226"
Sep 14 10:23:44.807274 ip-10-0-132-4 kubenswrapper[2563]: E0914 10:23:44.807248    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df643f9ce62433be3e7ca60117421bce05037c3dceabf1e9da073f62be987226\": container with ID starting with df643f9ce62433be3e7ca60117421bce05037c3dceabf1e9da073f62be987226 not found: ID does not exist" containerID="df643f9ce62433be3e7ca60117421bce05037c3dceabf1e9da073f62be987226"
Sep 14 10:23:44.807369 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:44.807280    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df643f9ce62433be3e7ca60117421bce05037c3dceabf1e9da073f62be987226"} err="failed to get container status \"df643f9ce62433be3e7ca60117421bce05037c3dceabf1e9da073f62be987226\": rpc error: code = NotFound desc = could not find container \"df643f9ce62433be3e7ca60117421bce05037c3dceabf1e9da073f62be987226\": container with ID starting with df643f9ce62433be3e7ca60117421bce05037c3dceabf1e9da073f62be987226 not found: ID does not exist"
Sep 14 10:23:44.819373 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:44.819345    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-82942479-kserve-85585595c5-kpvwq"]
Sep 14 10:23:44.822830 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:44.822797    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-82942479-kserve-85585595c5-kpvwq"]
Sep 14 10:23:45.482231 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:45.482195    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94cfd911-a61e-4d49-b1c0-e27f54e609e5" path="/var/lib/kubelet/pods/94cfd911-a61e-4d49-b1c0-e27f54e609e5/volumes"
Sep 14 10:23:50.229423 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:50.229381    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:23:50.330345 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:23:50.330315    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:24:17.359414 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:17.359332    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-86d8ccc4f4-xsck7_df7eded7-4c9c-4d49-9ae0-4cb3c8eeb136/authorino/0.log"
Sep 14 10:24:22.936578 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:22.936546    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-rlcpv_b419264c-1e5e-407a-9505-38c1b8a22721/server/0.log"
Sep 14 10:24:23.187063 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:23.186983    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84fdd9869c-4v5sx_a452960c-7230-4b45-9b8b-999b0123033d/copy-manifests/0.log"
Sep 14 10:24:23.195049 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:23.195014    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84fdd9869c-4v5sx_a452960c-7230-4b45-9b8b-999b0123033d/manager/0.log"
Sep 14 10:24:24.028015 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:24.027970    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29823000-b58d6_3f9f3e6b-0583-421c-90ad-c83e44e60580/cleanup/0.log"
Sep 14 10:24:24.140183 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:24.140150    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29823015-v5lw8_58c33d07-7ff3-469f-abc0-95328375ef7e/cleanup/0.log"
Sep 14 10:24:25.033180 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:25.033151    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-86d8ccc4f4-xsck7_df7eded7-4c9c-4d49-9ae0-4cb3c8eeb136/authorino/0.log"
Sep 14 10:24:26.082929 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:26.082896    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-lvspj_e1436e28-a6d7-42eb-a096-f0970a2145b4/istio-proxy/0.log"
Sep 14 10:24:26.316948 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:26.316916    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65d9bcd789-c4mnb_9dea6846-7ff5-4959-9d8c-2c7b04b85f60/kube-auth-proxy/0.log"
Sep 14 10:24:27.223799 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:27.223767    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-5bhm9_1a6aa52e-92c8-498f-a935-75ef0ea46e6a/main/0.log"
Sep 14 10:24:27.329638 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:27.329610    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-7vg96_c2f5a76a-2f1f-44b6-877b-943f7a78e4b8/main/0.log"
Sep 14 10:24:32.336115 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:32.336082    2563 scope.go:117] "RemoveContainer" containerID="a380450aab296c5820be9a412db508646299f50bdf8eda640bbecda11f775fd1"
Sep 14 10:24:32.346980 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:32.346958    2563 scope.go:117] "RemoveContainer" containerID="b5dee3a37aa7fac726948526f664978c30844916d916d30fcf9795fc0e57e53b"
Sep 14 10:24:33.784431 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:33.784397    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dhktf/must-gather-sgmrt"]
Sep 14 10:24:33.784795 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:33.784738    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c11f6996-1c88-499e-9903-7b230387a49f" containerName="main"
Sep 14 10:24:33.784795 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:33.784749    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c11f6996-1c88-499e-9903-7b230387a49f" containerName="main"
Sep 14 10:24:33.784795 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:33.784760    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94cfd911-a61e-4d49-b1c0-e27f54e609e5" containerName="main"
Sep 14 10:24:33.784795 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:33.784766    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="94cfd911-a61e-4d49-b1c0-e27f54e609e5" containerName="main"
Sep 14 10:24:33.784934 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:33.784831    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="94cfd911-a61e-4d49-b1c0-e27f54e609e5" containerName="main"
Sep 14 10:24:33.784934 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:33.784841    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c11f6996-1c88-499e-9903-7b230387a49f" containerName="main"
Sep 14 10:24:33.789381 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:33.789360    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dhktf/must-gather-sgmrt"
Sep 14 10:24:33.791911 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:33.791888    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dhktf\"/\"openshift-service-ca.crt\""
Sep 14 10:24:33.792080 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:33.791988    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-dhktf\"/\"default-dockercfg-7kn6x\""
Sep 14 10:24:33.792553 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:33.792539    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dhktf\"/\"kube-root-ca.crt\""
Sep 14 10:24:33.798938 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:33.798917    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dhktf/must-gather-sgmrt"]
Sep 14 10:24:33.801694 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:33.801670    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:24:33.883107 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:33.883073    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0d2bf709-c19e-4f1d-a208-16f0f76a74b1-must-gather-output\") pod \"must-gather-sgmrt\" (UID: \"0d2bf709-c19e-4f1d-a208-16f0f76a74b1\") " pod="openshift-must-gather-dhktf/must-gather-sgmrt"
Sep 14 10:24:33.883107 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:33.883114    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gq74\" (UniqueName: \"kubernetes.io/projected/0d2bf709-c19e-4f1d-a208-16f0f76a74b1-kube-api-access-7gq74\") pod \"must-gather-sgmrt\" (UID: \"0d2bf709-c19e-4f1d-a208-16f0f76a74b1\") " pod="openshift-must-gather-dhktf/must-gather-sgmrt"
Sep 14 10:24:33.907200 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:33.907167    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log" limit=1024
Sep 14 10:24:33.983592 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:33.983561    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0d2bf709-c19e-4f1d-a208-16f0f76a74b1-must-gather-output\") pod \"must-gather-sgmrt\" (UID: \"0d2bf709-c19e-4f1d-a208-16f0f76a74b1\") " pod="openshift-must-gather-dhktf/must-gather-sgmrt"
Sep 14 10:24:33.983733 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:33.983600    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gq74\" (UniqueName: \"kubernetes.io/projected/0d2bf709-c19e-4f1d-a208-16f0f76a74b1-kube-api-access-7gq74\") pod \"must-gather-sgmrt\" (UID: \"0d2bf709-c19e-4f1d-a208-16f0f76a74b1\") " pod="openshift-must-gather-dhktf/must-gather-sgmrt"
Sep 14 10:24:33.983964 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:33.983945    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0d2bf709-c19e-4f1d-a208-16f0f76a74b1-must-gather-output\") pod \"must-gather-sgmrt\" (UID: \"0d2bf709-c19e-4f1d-a208-16f0f76a74b1\") " pod="openshift-must-gather-dhktf/must-gather-sgmrt"
Sep 14 10:24:33.993571 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:33.993546    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gq74\" (UniqueName: \"kubernetes.io/projected/0d2bf709-c19e-4f1d-a208-16f0f76a74b1-kube-api-access-7gq74\") pod \"must-gather-sgmrt\" (UID: \"0d2bf709-c19e-4f1d-a208-16f0f76a74b1\") " pod="openshift-must-gather-dhktf/must-gather-sgmrt"
Sep 14 10:24:34.099683 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:34.099610    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dhktf/must-gather-sgmrt"
Sep 14 10:24:34.386111 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:34.386081    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dhktf/must-gather-sgmrt"]
Sep 14 10:24:34.387474 ip-10-0-132-4 kubenswrapper[2563]: W0914 10:24:34.387450    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d2bf709_c19e_4f1d_a208_16f0f76a74b1.slice/crio-a93b428f9e65ad37b30324bd5bd10e4835420a7ba7862da97484127e7059a941 WatchSource:0}: Error finding container a93b428f9e65ad37b30324bd5bd10e4835420a7ba7862da97484127e7059a941: Status 404 returned error can't find the container with id a93b428f9e65ad37b30324bd5bd10e4835420a7ba7862da97484127e7059a941
Sep 14 10:24:34.389409 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:34.389391    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 14 10:24:35.067218 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:35.067181    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dhktf/must-gather-sgmrt" event={"ID":"0d2bf709-c19e-4f1d-a208-16f0f76a74b1","Type":"ContainerStarted","Data":"a93b428f9e65ad37b30324bd5bd10e4835420a7ba7862da97484127e7059a941"}
Sep 14 10:24:36.076509 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:36.076462    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dhktf/must-gather-sgmrt" event={"ID":"0d2bf709-c19e-4f1d-a208-16f0f76a74b1","Type":"ContainerStarted","Data":"7f86c8f647f5901892ffc2d97eca810b211e8753d37a1218a031da748ff7d0e6"}
Sep 14 10:24:36.076946 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:36.076522    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dhktf/must-gather-sgmrt" event={"ID":"0d2bf709-c19e-4f1d-a208-16f0f76a74b1","Type":"ContainerStarted","Data":"c005639b00072a2977a63ee8562bda26f6ce6cec4225b352213c28831448c180"}
Sep 14 10:24:36.094168 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:36.094109    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-dhktf/must-gather-sgmrt" podStartSLOduration=2.233455464 podStartE2EDuration="3.094090174s" totalImagesPullingTime="860.63471ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:24:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:24:36.092094999 +0000 UTC m=+2465.131769603" watchObservedRunningTime="2026-09-14 10:24:36.094090174 +0000 UTC m=+2465.133764837"
Sep 14 10:24:36.912344 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:36.912296    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-krg4v_97284aa2-9a80-495f-8b99-2ff7cec913da/global-pull-secret-syncer/0.log"
Sep 14 10:24:37.033824 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:37.033789    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-cskmj_66c47172-17e5-4985-84a8-987c216dd4ec/konnectivity-agent/0.log"
Sep 14 10:24:37.157588 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:37.157552    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-4.ec2.internal_812604c7944c05f98e0873536f91322d/haproxy/0.log"
Sep 14 10:24:41.869115 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:41.869077    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-86d8ccc4f4-xsck7_df7eded7-4c9c-4d49-9ae0-4cb3c8eeb136/authorino/0.log"
Sep 14 10:24:43.800036 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:43.799356    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-7tpzw_161e5bed-e6f8-4c08-8c69-857064cc8c4f/cluster-monitoring-operator/0.log"
Sep 14 10:24:43.833902 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:43.833850    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-v8jjn_935ba669-ed70-406a-a645-054a8db8b24b/kube-state-metrics/0.log"
Sep 14 10:24:43.860018 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:43.859966    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-v8jjn_935ba669-ed70-406a-a645-054a8db8b24b/kube-rbac-proxy-main/0.log"
Sep 14 10:24:43.889114 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:43.889081    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-v8jjn_935ba669-ed70-406a-a645-054a8db8b24b/kube-rbac-proxy-self/0.log"
Sep 14 10:24:43.925397 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:43.925362    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-65fb75f47b-lx68z_2ecb5cff-e201-48cd-8837-a432af591332/metrics-server/0.log"
Sep 14 10:24:43.959843 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:43.959804    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-n75c8_81e3b875-1204-4d99-b431-8ded39ac08e8/monitoring-plugin/0.log"
Sep 14 10:24:44.185426 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:44.185333    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zkffp_dabf7719-e039-48a1-8150-f100f09e2d4a/node-exporter/0.log"
Sep 14 10:24:44.215957 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:44.215919    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zkffp_dabf7719-e039-48a1-8150-f100f09e2d4a/kube-rbac-proxy/0.log"
Sep 14 10:24:44.252051 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:44.251993    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zkffp_dabf7719-e039-48a1-8150-f100f09e2d4a/init-textfile/0.log"
Sep 14 10:24:44.298579 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:44.298551    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-ldtq5_698195e3-0a13-4176-ab4c-f8167e92eefb/kube-rbac-proxy-main/0.log"
Sep 14 10:24:44.342760 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:44.342610    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-ldtq5_698195e3-0a13-4176-ab4c-f8167e92eefb/kube-rbac-proxy-self/0.log"
Sep 14 10:24:44.381037 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:44.380990    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-ldtq5_698195e3-0a13-4176-ab4c-f8167e92eefb/openshift-state-metrics/0.log"
Sep 14 10:24:47.026029 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:47.025976    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dhktf/perf-node-gather-daemonset-bdnpj"]
Sep 14 10:24:47.032391 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:47.032355    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dhktf/perf-node-gather-daemonset-bdnpj"
Sep 14 10:24:47.040377 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:47.040345    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dhktf/perf-node-gather-daemonset-bdnpj"]
Sep 14 10:24:47.107423 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:47.107355    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a627e46a-f059-416d-ab29-f2819ec3d3ee-sys\") pod \"perf-node-gather-daemonset-bdnpj\" (UID: \"a627e46a-f059-416d-ab29-f2819ec3d3ee\") " pod="openshift-must-gather-dhktf/perf-node-gather-daemonset-bdnpj"
Sep 14 10:24:47.107648 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:47.107447    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58vpd\" (UniqueName: \"kubernetes.io/projected/a627e46a-f059-416d-ab29-f2819ec3d3ee-kube-api-access-58vpd\") pod \"perf-node-gather-daemonset-bdnpj\" (UID: \"a627e46a-f059-416d-ab29-f2819ec3d3ee\") " pod="openshift-must-gather-dhktf/perf-node-gather-daemonset-bdnpj"
Sep 14 10:24:47.107648 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:47.107483    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a627e46a-f059-416d-ab29-f2819ec3d3ee-lib-modules\") pod \"perf-node-gather-daemonset-bdnpj\" (UID: \"a627e46a-f059-416d-ab29-f2819ec3d3ee\") " pod="openshift-must-gather-dhktf/perf-node-gather-daemonset-bdnpj"
Sep 14 10:24:47.107648 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:47.107616    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a627e46a-f059-416d-ab29-f2819ec3d3ee-podres\") pod \"perf-node-gather-daemonset-bdnpj\" (UID: \"a627e46a-f059-416d-ab29-f2819ec3d3ee\") " pod="openshift-must-gather-dhktf/perf-node-gather-daemonset-bdnpj"
Sep 14 10:24:47.107827 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:47.107674    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a627e46a-f059-416d-ab29-f2819ec3d3ee-proc\") pod \"perf-node-gather-daemonset-bdnpj\" (UID: \"a627e46a-f059-416d-ab29-f2819ec3d3ee\") " pod="openshift-must-gather-dhktf/perf-node-gather-daemonset-bdnpj"
Sep 14 10:24:47.208400 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:47.208363    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a627e46a-f059-416d-ab29-f2819ec3d3ee-podres\") pod \"perf-node-gather-daemonset-bdnpj\" (UID: \"a627e46a-f059-416d-ab29-f2819ec3d3ee\") " pod="openshift-must-gather-dhktf/perf-node-gather-daemonset-bdnpj"
Sep 14 10:24:47.208400 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:47.208413    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a627e46a-f059-416d-ab29-f2819ec3d3ee-proc\") pod \"perf-node-gather-daemonset-bdnpj\" (UID: \"a627e46a-f059-416d-ab29-f2819ec3d3ee\") " pod="openshift-must-gather-dhktf/perf-node-gather-daemonset-bdnpj"
Sep 14 10:24:47.208641 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:47.208457    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a627e46a-f059-416d-ab29-f2819ec3d3ee-sys\") pod \"perf-node-gather-daemonset-bdnpj\" (UID: \"a627e46a-f059-416d-ab29-f2819ec3d3ee\") " pod="openshift-must-gather-dhktf/perf-node-gather-daemonset-bdnpj"
Sep 14 10:24:47.208641 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:47.208496    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58vpd\" (UniqueName: \"kubernetes.io/projected/a627e46a-f059-416d-ab29-f2819ec3d3ee-kube-api-access-58vpd\") pod \"perf-node-gather-daemonset-bdnpj\" (UID: \"a627e46a-f059-416d-ab29-f2819ec3d3ee\") " pod="openshift-must-gather-dhktf/perf-node-gather-daemonset-bdnpj"
Sep 14 10:24:47.208641 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:47.208520    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a627e46a-f059-416d-ab29-f2819ec3d3ee-lib-modules\") pod \"perf-node-gather-daemonset-bdnpj\" (UID: \"a627e46a-f059-416d-ab29-f2819ec3d3ee\") " pod="openshift-must-gather-dhktf/perf-node-gather-daemonset-bdnpj"
Sep 14 10:24:47.208641 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:47.208575    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a627e46a-f059-416d-ab29-f2819ec3d3ee-podres\") pod \"perf-node-gather-daemonset-bdnpj\" (UID: \"a627e46a-f059-416d-ab29-f2819ec3d3ee\") " pod="openshift-must-gather-dhktf/perf-node-gather-daemonset-bdnpj"
Sep 14 10:24:47.208641 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:47.208603    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a627e46a-f059-416d-ab29-f2819ec3d3ee-proc\") pod \"perf-node-gather-daemonset-bdnpj\" (UID: \"a627e46a-f059-416d-ab29-f2819ec3d3ee\") " pod="openshift-must-gather-dhktf/perf-node-gather-daemonset-bdnpj"
Sep 14 10:24:47.208641 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:47.208639    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a627e46a-f059-416d-ab29-f2819ec3d3ee-sys\") pod \"perf-node-gather-daemonset-bdnpj\" (UID: \"a627e46a-f059-416d-ab29-f2819ec3d3ee\") " pod="openshift-must-gather-dhktf/perf-node-gather-daemonset-bdnpj"
Sep 14 10:24:47.208849 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:47.208654    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a627e46a-f059-416d-ab29-f2819ec3d3ee-lib-modules\") pod \"perf-node-gather-daemonset-bdnpj\" (UID: \"a627e46a-f059-416d-ab29-f2819ec3d3ee\") " pod="openshift-must-gather-dhktf/perf-node-gather-daemonset-bdnpj"
Sep 14 10:24:47.217901 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:47.217860    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58vpd\" (UniqueName: \"kubernetes.io/projected/a627e46a-f059-416d-ab29-f2819ec3d3ee-kube-api-access-58vpd\") pod \"perf-node-gather-daemonset-bdnpj\" (UID: \"a627e46a-f059-416d-ab29-f2819ec3d3ee\") " pod="openshift-must-gather-dhktf/perf-node-gather-daemonset-bdnpj"
Sep 14 10:24:47.349792 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:47.349482    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dhktf/perf-node-gather-daemonset-bdnpj"
Sep 14 10:24:47.654647 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:47.654615    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dhktf/perf-node-gather-daemonset-bdnpj"]
Sep 14 10:24:47.656997 ip-10-0-132-4 kubenswrapper[2563]: W0914 10:24:47.656964    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda627e46a_f059_416d_ab29_f2819ec3d3ee.slice/crio-bd7a1f2c65e08e0efeba22381d4267eaf783af7ced8f2c28a84f098bdf6d0ed5 WatchSource:0}: Error finding container bd7a1f2c65e08e0efeba22381d4267eaf783af7ced8f2c28a84f098bdf6d0ed5: Status 404 returned error can't find the container with id bd7a1f2c65e08e0efeba22381d4267eaf783af7ced8f2c28a84f098bdf6d0ed5
Sep 14 10:24:48.181652 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:48.181605    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dhktf/perf-node-gather-daemonset-bdnpj" event={"ID":"a627e46a-f059-416d-ab29-f2819ec3d3ee","Type":"ContainerStarted","Data":"645cd1d33c46508b3a7e56c2e570c0460a370d13369e58cffb66128a5736e9b0"}
Sep 14 10:24:48.182182 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:48.181680    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dhktf/perf-node-gather-daemonset-bdnpj" event={"ID":"a627e46a-f059-416d-ab29-f2819ec3d3ee","Type":"ContainerStarted","Data":"bd7a1f2c65e08e0efeba22381d4267eaf783af7ced8f2c28a84f098bdf6d0ed5"}
Sep 14 10:24:48.182606 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:48.182577    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-dhktf/perf-node-gather-daemonset-bdnpj"
Sep 14 10:24:49.678983 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:49.678943    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mklrw_ec1015e1-7070-4f02-8f4e-2fd5d395c4e9/dns/0.log"
Sep 14 10:24:49.702578 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:49.702543    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mklrw_ec1015e1-7070-4f02-8f4e-2fd5d395c4e9/kube-rbac-proxy/0.log"
Sep 14 10:24:49.807935 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:49.807896    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-f5jts_da4b31da-fcbb-4132-bc49-1ecd91bb65bb/dns-node-resolver/0.log"
Sep 14 10:24:50.397141 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:50.397106    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-544f748fdb-5pr6x_f610700e-43a1-443a-a589-3420e9ce4c91/registry/0.log"
Sep 14 10:24:50.454182 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:50.454143    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mq9n4_07d5e1df-7d96-401d-b606-245e4b1f3ae5/node-ca/0.log"
Sep 14 10:24:51.524574 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:51.524542    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-lvspj_e1436e28-a6d7-42eb-a096-f0970a2145b4/istio-proxy/0.log"
Sep 14 10:24:52.346071 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:52.346038    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65d9bcd789-c4mnb_9dea6846-7ff5-4959-9d8c-2c7b04b85f60/kube-auth-proxy/0.log"
Sep 14 10:24:53.490278 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:53.490246    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-f7d9v_4a0bee38-c70e-420f-82f8-ff4f0c42e0a6/serve-healthcheck-canary/0.log"
Sep 14 10:24:54.212198 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:54.212162    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tvs2c_bf286afe-8eb2-4e87-bc9c-5d8646662e58/kube-rbac-proxy/0.log"
Sep 14 10:24:54.237780 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:54.237748    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tvs2c_bf286afe-8eb2-4e87-bc9c-5d8646662e58/exporter/0.log"
Sep 14 10:24:54.264137 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:54.264102    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tvs2c_bf286afe-8eb2-4e87-bc9c-5d8646662e58/extractor/0.log"
Sep 14 10:24:55.207032 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:55.206978    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-dhktf/perf-node-gather-daemonset-bdnpj"
Sep 14 10:24:55.237606 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:55.237536    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-dhktf/perf-node-gather-daemonset-bdnpj" podStartSLOduration=8.237515436 podStartE2EDuration="8.237515436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 10:24:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:24:48.202515187 +0000 UTC m=+2477.242189809" watchObservedRunningTime="2026-09-14 10:24:55.237515436 +0000 UTC m=+2484.277190055"
Sep 14 10:24:56.835899 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:56.835869    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-rlcpv_b419264c-1e5e-407a-9505-38c1b8a22721/server/0.log"
Sep 14 10:24:56.998982 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:56.998940    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84fdd9869c-4v5sx_a452960c-7230-4b45-9b8b-999b0123033d/manager/0.log"
Sep 14 10:24:57.030213 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:57.030181    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84fdd9869c-4v5sx_a452960c-7230-4b45-9b8b-999b0123033d/copy-manifests/0.log"
Sep 14 10:24:57.922246 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:57.922147    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-lvspj_e1436e28-a6d7-42eb-a096-f0970a2145b4/istio-proxy/0.log"
Sep 14 10:24:58.720765 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:58.720683    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5d6ddf947f-kmx7m_0521996d-31be-49c0-84ca-a3ba08d80d71/manager/0.log"
Sep 14 10:24:58.743419 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:24:58.743389    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65d9bcd789-c4mnb_9dea6846-7ff5-4959-9d8c-2c7b04b85f60/kube-auth-proxy/0.log"
Sep 14 10:25:00.439557 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:25:00.439518    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-lvspj_e1436e28-a6d7-42eb-a096-f0970a2145b4/istio-proxy/0.log"
Sep 14 10:25:01.269905 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:25:01.269872    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65d9bcd789-c4mnb_9dea6846-7ff5-4959-9d8c-2c7b04b85f60/kube-auth-proxy/0.log"
Sep 14 10:25:04.026035 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:25:04.025989    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-rqpdl_f2e9bff6-a63c-4334-897a-1fe8d29ba23a/migrator/0.log"
Sep 14 10:25:04.051130 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:25:04.051096    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-rqpdl_f2e9bff6-a63c-4334-897a-1fe8d29ba23a/graceful-termination/0.log"
Sep 14 10:25:05.816309 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:25:05.816275    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7tm8l_20d9618f-664c-4cae-bcc2-4b00ba45acc6/kube-multus-additional-cni-plugins/0.log"
Sep 14 10:25:05.841629 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:25:05.841595    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7tm8l_20d9618f-664c-4cae-bcc2-4b00ba45acc6/egress-router-binary-copy/0.log"
Sep 14 10:25:05.867934 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:25:05.867907    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7tm8l_20d9618f-664c-4cae-bcc2-4b00ba45acc6/cni-plugins/0.log"
Sep 14 10:25:05.892254 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:25:05.892204    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7tm8l_20d9618f-664c-4cae-bcc2-4b00ba45acc6/bond-cni-plugin/0.log"
Sep 14 10:25:05.915953 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:25:05.915923    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7tm8l_20d9618f-664c-4cae-bcc2-4b00ba45acc6/routeoverride-cni/0.log"
Sep 14 10:25:05.938687 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:25:05.938654    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7tm8l_20d9618f-664c-4cae-bcc2-4b00ba45acc6/whereabouts-cni-bincopy/0.log"
Sep 14 10:25:05.962047 ip-10-0-132-4 kubenswrapper[2563]: I0914 10:25:05.962014    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7tm8l_20d9618f-664c-4cae-bcc2-4b00ba45acc6/whereabouts-cni/0.log"