Sep 17 13:53:47.066065 ip-10-0-129-158 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 17 13:53:47.066076 ip-10-0-129-158 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 17 13:53:47.066085 ip-10-0-129-158 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 17 13:53:47.066436 ip-10-0-129-158 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 17 13:53:57.205865 ip-10-0-129-158 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 17 13:53:57.205881 ip-10-0-129-158 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot dbd3ec68d6d046fa931da494c2b310f3 --
Sep 17 13:55:30.952721 ip-10-0-129-158 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 13:55:31.376829 ip-10-0-129-158 kubenswrapper[2560]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 13:55:31.376829 ip-10-0-129-158 kubenswrapper[2560]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 13:55:31.376829 ip-10-0-129-158 kubenswrapper[2560]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 13:55:31.376829 ip-10-0-129-158 kubenswrapper[2560]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 13:55:31.376829 ip-10-0-129-158 kubenswrapper[2560]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 13:55:31.377764 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.377690    2560 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 13:55:31.383239 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383224    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 13:55:31.383239 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383239    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 13:55:31.383301 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383243    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 13:55:31.383301 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383247    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 13:55:31.383301 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383251    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 13:55:31.383301 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383260    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 13:55:31.383301 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383264    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 13:55:31.383301 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383266    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 13:55:31.383301 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383269    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 13:55:31.383301 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383271    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 13:55:31.383301 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383274    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 13:55:31.383301 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383277    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 13:55:31.383301 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383279    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 13:55:31.383301 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383282    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 13:55:31.383301 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383284    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 13:55:31.383301 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383287    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 13:55:31.383301 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383289    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 13:55:31.383301 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383292    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 13:55:31.383301 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383294    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 13:55:31.383301 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383296    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 13:55:31.383301 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383299    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 13:55:31.383751 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383302    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 13:55:31.383751 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383305    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 13:55:31.383751 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383307    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 13:55:31.383751 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383310    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 13:55:31.383751 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383313    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 13:55:31.383751 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383315    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 13:55:31.383751 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383318    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 13:55:31.383751 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383320    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 13:55:31.383751 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383323    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 13:55:31.383751 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383325    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 13:55:31.383751 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383329    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 13:55:31.383751 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383333    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 13:55:31.383751 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383335    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 13:55:31.383751 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383338    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 13:55:31.383751 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383341    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 13:55:31.383751 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383344    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 13:55:31.383751 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383347    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 13:55:31.383751 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383350    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 13:55:31.383751 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383352    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 13:55:31.384201 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383355    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 13:55:31.384201 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383357    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 13:55:31.384201 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383360    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 13:55:31.384201 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383362    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 13:55:31.384201 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383364    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 13:55:31.384201 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383367    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 13:55:31.384201 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383369    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 13:55:31.384201 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383372    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 13:55:31.384201 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383374    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 13:55:31.384201 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383377    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 13:55:31.384201 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383380    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 13:55:31.384201 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383384    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 13:55:31.384201 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383387    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 13:55:31.384201 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383389    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 13:55:31.384201 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383392    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 13:55:31.384201 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383395    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 13:55:31.384201 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383397    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 13:55:31.384201 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383400    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 13:55:31.384201 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383402    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 13:55:31.384201 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383406    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 13:55:31.384720 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383408    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 13:55:31.384720 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383410    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 13:55:31.384720 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383413    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 13:55:31.384720 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383415    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 13:55:31.384720 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383418    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 13:55:31.384720 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383420    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 13:55:31.384720 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383422    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 13:55:31.384720 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383425    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 13:55:31.384720 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383427    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 13:55:31.384720 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383431    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 13:55:31.384720 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383434    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 13:55:31.384720 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383438    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 13:55:31.384720 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383440    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 13:55:31.384720 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383443    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 13:55:31.384720 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383445    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 13:55:31.384720 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383448    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 13:55:31.384720 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383450    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 13:55:31.384720 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383453    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 13:55:31.384720 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383456    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 13:55:31.384720 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383468    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 13:55:31.385204 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383471    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 13:55:31.385204 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383473    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 13:55:31.385204 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383476    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 13:55:31.385204 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383479    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 13:55:31.385204 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383481    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 13:55:31.385204 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383483    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 13:55:31.385204 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383486    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 13:55:31.385204 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383489    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 13:55:31.385204 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383491    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 13:55:31.385204 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383494    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 13:55:31.385204 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383497    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 13:55:31.385204 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383499    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 13:55:31.385204 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383502    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 13:55:31.385204 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383504    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 13:55:31.385204 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383507    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 13:55:31.385204 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383509    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 13:55:31.385204 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383512    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 13:55:31.385204 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383514    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 13:55:31.385204 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.383516    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 13:55:31.385204 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384175    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 13:55:31.385734 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384182    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 13:55:31.385734 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384185    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 13:55:31.385734 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384188    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 13:55:31.385734 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384191    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 13:55:31.385734 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384194    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 13:55:31.385734 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384198    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 13:55:31.385734 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384201    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 13:55:31.385734 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384204    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 13:55:31.385734 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384207    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 13:55:31.385734 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384211    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 13:55:31.385734 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384215    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 13:55:31.385734 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384217    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 13:55:31.385734 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384220    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 13:55:31.385734 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384223    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 13:55:31.385734 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384225    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 13:55:31.385734 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384228    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 13:55:31.385734 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384238    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 13:55:31.385734 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384242    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 13:55:31.385734 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384245    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 13:55:31.385734 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384248    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 13:55:31.386222 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384251    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 13:55:31.386222 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384253    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 13:55:31.386222 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384256    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 13:55:31.386222 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384258    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 13:55:31.386222 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384261    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 13:55:31.386222 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384263    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 13:55:31.386222 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384266    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 13:55:31.386222 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384287    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 13:55:31.386222 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384289    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 13:55:31.386222 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384292    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 13:55:31.386222 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384295    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 13:55:31.386222 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384298    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 13:55:31.386222 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384301    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 13:55:31.386222 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384304    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 13:55:31.386222 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384307    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 13:55:31.386222 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384310    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 13:55:31.386222 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384312    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 13:55:31.386222 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384316    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 13:55:31.386222 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384318    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 13:55:31.386684 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384321    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 13:55:31.386684 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384323    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 13:55:31.386684 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384326    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 13:55:31.386684 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384328    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 13:55:31.386684 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384331    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 13:55:31.386684 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384333    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 13:55:31.386684 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384336    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 13:55:31.386684 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384339    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 13:55:31.386684 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384342    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 13:55:31.386684 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384344    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 13:55:31.386684 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384347    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 13:55:31.386684 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384350    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 13:55:31.386684 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384353    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 13:55:31.386684 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384356    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 13:55:31.386684 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384358    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 13:55:31.386684 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384361    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 13:55:31.386684 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384365    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 13:55:31.386684 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384369    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 13:55:31.386684 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384372    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 13:55:31.386684 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384375    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 13:55:31.387151 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384377    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 13:55:31.387151 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384380    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 13:55:31.387151 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384382    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 13:55:31.387151 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384385    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 13:55:31.387151 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384387    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 13:55:31.387151 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384392    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 13:55:31.387151 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384395    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 13:55:31.387151 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384397    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 13:55:31.387151 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384400    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 13:55:31.387151 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384403    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 13:55:31.387151 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384405    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 13:55:31.387151 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384408    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 13:55:31.387151 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384411    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 13:55:31.387151 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384414    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 13:55:31.387151 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384416    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 13:55:31.387151 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384418    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 13:55:31.387151 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384421    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 13:55:31.387151 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384423    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 13:55:31.387151 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384426    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 13:55:31.387151 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384428    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 13:55:31.387639 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384431    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 13:55:31.387639 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384434    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 13:55:31.387639 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384437    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 13:55:31.387639 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384441    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 13:55:31.387639 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384444    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 13:55:31.387639 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384447    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 13:55:31.387639 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384450    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 13:55:31.387639 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384453    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 13:55:31.387639 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384456    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 13:55:31.387639 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384472    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 13:55:31.387639 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384474    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 13:55:31.387639 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384477    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 13:55:31.387639 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384480    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 13:55:31.387639 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384482    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 13:55:31.387639 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384485    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 13:55:31.387639 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384488    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 13:55:31.387639 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384490    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 13:55:31.387639 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384497    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 13:55:31.387639 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384499    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 13:55:31.388085 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384841    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 13:55:31.388085 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384845    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 13:55:31.388085 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384848    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 13:55:31.388085 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384851    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 13:55:31.388085 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384854    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 13:55:31.388085 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384857    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 13:55:31.388085 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384860    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 13:55:31.388085 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384862    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 13:55:31.388085 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384865    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 13:55:31.388085 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384867    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 13:55:31.388085 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384870    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 13:55:31.388085 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384873    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 13:55:31.388085 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384875    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 13:55:31.388085 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384878    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 13:55:31.388085 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384880    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 13:55:31.388085 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384883    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 13:55:31.388085 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384886    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 13:55:31.388085 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384888    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 13:55:31.388085 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384891    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 13:55:31.388085 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384894    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 13:55:31.388604 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384896    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 13:55:31.388604 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384899    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 13:55:31.388604 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384902    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 13:55:31.388604 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384905    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 13:55:31.388604 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384907    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 13:55:31.388604 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384910    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 13:55:31.388604 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384914    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 13:55:31.388604 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384918    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 13:55:31.388604 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384921    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 13:55:31.388604 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384924    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 13:55:31.388604 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384928    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 13:55:31.388604 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384931    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 13:55:31.388604 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384934    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 13:55:31.388604 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384936    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 13:55:31.388604 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384939    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 13:55:31.388604 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384943    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 13:55:31.388604 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384948    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 13:55:31.388604 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384952    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 13:55:31.389043 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384955    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 13:55:31.389043 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384958    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 13:55:31.389043 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384961    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 13:55:31.389043 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384964    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 13:55:31.389043 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384966    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 13:55:31.389043 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384969    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 13:55:31.389043 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384971    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 13:55:31.389043 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384974    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 13:55:31.389043 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384976    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 13:55:31.389043 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384979    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 13:55:31.389043 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384981    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 13:55:31.389043 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384984    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 13:55:31.389043 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384986    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 13:55:31.389043 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384989    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 13:55:31.389043 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384991    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 13:55:31.389043 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384994    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 13:55:31.389043 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384996    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 13:55:31.389043 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.384999    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 13:55:31.389043 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385001    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 13:55:31.389043 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385003    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 13:55:31.389556 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385006    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 13:55:31.389556 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385008    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 13:55:31.389556 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385011    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 13:55:31.389556 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385013    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 13:55:31.389556 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385018    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 13:55:31.389556 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385020    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 13:55:31.389556 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385023    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 13:55:31.389556 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385025    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 13:55:31.389556 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385027    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 13:55:31.389556 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385030    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 13:55:31.389556 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385032    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 13:55:31.389556 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385035    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 13:55:31.389556 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385038    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 13:55:31.389556 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385041    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 13:55:31.389556 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385043    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 13:55:31.389556 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385046    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 13:55:31.389556 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385048    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 13:55:31.389556 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385051    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 13:55:31.389556 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385053    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 13:55:31.390002 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385055    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 13:55:31.390002 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385058    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 13:55:31.390002 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385061    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 13:55:31.390002 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385065    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 13:55:31.390002 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385067    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 13:55:31.390002 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385070    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 13:55:31.390002 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385072    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 13:55:31.390002 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385075    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 13:55:31.390002 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385077    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 13:55:31.390002 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385080    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 13:55:31.390002 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385083    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 13:55:31.390002 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385085    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 13:55:31.390002 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385088    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 13:55:31.390002 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385090    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 13:55:31.390002 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385092    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 13:55:31.390002 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385095    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 13:55:31.390002 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385097    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 13:55:31.390002 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385101    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 13:55:31.390002 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385103    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 13:55:31.390002 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385106    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 13:55:31.390527 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385108    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 13:55:31.390527 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385111    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 13:55:31.390527 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385182    2560 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 13:55:31.390527 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385212    2560 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 13:55:31.390527 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385217    2560 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 13:55:31.390527 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385222    2560 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 13:55:31.390527 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385227    2560 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 13:55:31.390527 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385230    2560 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 13:55:31.390527 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385235    2560 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 13:55:31.390527 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385239    2560 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 13:55:31.390527 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385242    2560 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 13:55:31.390527 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385246    2560 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 13:55:31.390527 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385249    2560 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 13:55:31.390527 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385252    2560 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 13:55:31.390527 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385256    2560 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 13:55:31.390527 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385259    2560 flags.go:64] FLAG: --cgroup-root=""
Sep 17 13:55:31.390527 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385262    2560 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 13:55:31.390527 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385265    2560 flags.go:64] FLAG: --client-ca-file=""
Sep 17 13:55:31.390527 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385268    2560 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 13:55:31.390527 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385271    2560 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 13:55:31.390527 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385275    2560 flags.go:64] FLAG: --cluster-domain=""
Sep 17 13:55:31.390527 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385278    2560 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 13:55:31.390527 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385282    2560 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 13:55:31.390527 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385285    2560 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 13:55:31.391173 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385288    2560 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 13:55:31.391173 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385292    2560 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 13:55:31.391173 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385294    2560 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 13:55:31.391173 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385298    2560 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 13:55:31.391173 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385301    2560 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 13:55:31.391173 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385306    2560 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 13:55:31.391173 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385308    2560 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 13:55:31.391173 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385311    2560 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 13:55:31.391173 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385314    2560 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 13:55:31.391173 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385317    2560 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 13:55:31.391173 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385321    2560 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 13:55:31.391173 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385324    2560 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 13:55:31.391173 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385326    2560 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 13:55:31.391173 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385329    2560 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 13:55:31.391173 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385332    2560 flags.go:64] FLAG: --enable-server="true"
Sep 17 13:55:31.391173 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385336    2560 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 13:55:31.391173 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385340    2560 flags.go:64] FLAG: --event-burst="100"
Sep 17 13:55:31.391173 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385343    2560 flags.go:64] FLAG: --event-qps="50"
Sep 17 13:55:31.391173 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385345    2560 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 13:55:31.391173 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385348    2560 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 13:55:31.391173 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385351    2560 flags.go:64] FLAG: --eviction-hard=""
Sep 17 13:55:31.391173 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385355    2560 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 13:55:31.391173 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385357    2560 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 13:55:31.391173 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385361    2560 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 13:55:31.391173 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385364    2560 flags.go:64] FLAG: --eviction-soft=""
Sep 17 13:55:31.391782 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385367    2560 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 13:55:31.391782 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385370    2560 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 13:55:31.391782 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385373    2560 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 13:55:31.391782 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385375    2560 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 13:55:31.391782 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385378    2560 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 13:55:31.391782 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385381    2560 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 13:55:31.391782 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385384    2560 flags.go:64] FLAG: --feature-gates=""
Sep 17 13:55:31.391782 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385388    2560 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 13:55:31.391782 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385391    2560 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 13:55:31.391782 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385394    2560 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 13:55:31.391782 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385397    2560 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 13:55:31.391782 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385400    2560 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 13:55:31.391782 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385404    2560 flags.go:64] FLAG: --help="false"
Sep 17 13:55:31.391782 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385406    2560 flags.go:64] FLAG: --hostname-override="ip-10-0-129-158.ec2.internal"
Sep 17 13:55:31.391782 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385410    2560 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 13:55:31.391782 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385413    2560 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 13:55:31.391782 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385416    2560 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 13:55:31.391782 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385419    2560 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 13:55:31.391782 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385423    2560 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 13:55:31.391782 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385425    2560 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 13:55:31.391782 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385428    2560 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 13:55:31.391782 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385431    2560 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 13:55:31.391782 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385435    2560 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 13:55:31.392376 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385438    2560 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 13:55:31.392376 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385441    2560 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 13:55:31.392376 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385444    2560 flags.go:64] FLAG: --kube-reserved=""
Sep 17 13:55:31.392376 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385447    2560 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 13:55:31.392376 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385450    2560 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 13:55:31.392376 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385453    2560 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 13:55:31.392376 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385456    2560 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 13:55:31.392376 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385474    2560 flags.go:64] FLAG: --lock-file=""
Sep 17 13:55:31.392376 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385479    2560 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 13:55:31.392376 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385483    2560 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 13:55:31.392376 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385488    2560 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 13:55:31.392376 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385494    2560 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 13:55:31.392376 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385508    2560 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 13:55:31.392376 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385512    2560 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 13:55:31.392376 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385515    2560 flags.go:64] FLAG: --logging-format="text"
Sep 17 13:55:31.392376 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385517    2560 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 13:55:31.392376 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385521    2560 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 13:55:31.392376 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385523    2560 flags.go:64] FLAG: --manifest-url=""
Sep 17 13:55:31.392376 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385526    2560 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 13:55:31.392376 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385531    2560 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 13:55:31.392376 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385534    2560 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 13:55:31.392376 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385539    2560 flags.go:64] FLAG: --max-pods="110"
Sep 17 13:55:31.392376 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385542    2560 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 13:55:31.392376 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385545    2560 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 13:55:31.392376 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385548    2560 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 13:55:31.393040 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385551    2560 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 13:55:31.393040 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385554    2560 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 13:55:31.393040 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385557    2560 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 13:55:31.393040 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385559    2560 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 13:55:31.393040 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385566    2560 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 13:55:31.393040 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385569    2560 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 13:55:31.393040 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385572    2560 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 13:55:31.393040 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385575    2560 flags.go:64] FLAG: --pod-cidr=""
Sep 17 13:55:31.393040 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385578    2560 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 17 13:55:31.393040 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385583    2560 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 13:55:31.393040 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385586    2560 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 13:55:31.393040 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385589    2560 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 13:55:31.393040 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385592    2560 flags.go:64] FLAG: --port="10250"
Sep 17 13:55:31.393040 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385595    2560 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 13:55:31.393040 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385598    2560 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-097b69b49947028f3"
Sep 17 13:55:31.393040 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385601    2560 flags.go:64] FLAG: --qos-reserved=""
Sep 17 13:55:31.393040 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385604    2560 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 13:55:31.393040 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385606    2560 flags.go:64] FLAG: --register-node="true"
Sep 17 13:55:31.393040 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385609    2560 flags.go:64] FLAG: --register-with-taints=""
Sep 17 13:55:31.393040 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385613    2560 flags.go:64] FLAG: --registry-burst="10"
Sep 17 13:55:31.393040 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385615    2560 flags.go:64] FLAG: --registry-qps="5"
Sep 17 13:55:31.393040 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385618    2560 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 13:55:31.393040 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385621    2560 flags.go:64] FLAG: --reserved-memory=""
Sep 17 13:55:31.393040 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385624    2560 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 13:55:31.393040 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385627    2560 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 13:55:31.393670 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385630    2560 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 13:55:31.393670 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385633    2560 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 13:55:31.393670 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385635    2560 flags.go:64] FLAG: --runonce="false"
Sep 17 13:55:31.393670 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385639    2560 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 13:55:31.393670 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385642    2560 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 13:55:31.393670 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385645    2560 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 13:55:31.393670 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385648    2560 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 13:55:31.393670 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385651    2560 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 13:55:31.393670 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385653    2560 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 13:55:31.393670 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385657    2560 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 13:55:31.393670 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385660    2560 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 13:55:31.393670 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385662    2560 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 13:55:31.393670 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385665    2560 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 13:55:31.393670 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385668    2560 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 13:55:31.393670 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385670    2560 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 13:55:31.393670 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385674    2560 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 13:55:31.393670 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385677    2560 flags.go:64] FLAG: --system-cgroups=""
Sep 17 13:55:31.393670 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385680    2560 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 13:55:31.393670 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385684    2560 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 13:55:31.393670 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385687    2560 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 13:55:31.393670 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385690    2560 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 13:55:31.393670 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385694    2560 flags.go:64] FLAG: --tls-min-version=""
Sep 17 13:55:31.393670 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385698    2560 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 13:55:31.393670 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385700    2560 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 13:55:31.393670 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385703    2560 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 13:55:31.394451 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385706    2560 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 13:55:31.394451 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385709    2560 flags.go:64] FLAG: --v="2"
Sep 17 13:55:31.394451 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385712    2560 flags.go:64] FLAG: --version="false"
Sep 17 13:55:31.394451 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385716    2560 flags.go:64] FLAG: --vmodule=""
Sep 17 13:55:31.394451 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385720    2560 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 13:55:31.394451 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.385723    2560 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 13:55:31.394451 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385827    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 13:55:31.394451 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385831    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 13:55:31.394451 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385833    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 13:55:31.394451 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385836    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 13:55:31.394451 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385839    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 13:55:31.394451 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385842    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 13:55:31.394451 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385845    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 13:55:31.394451 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385847    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 13:55:31.394451 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385850    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 13:55:31.394451 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385852    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 13:55:31.394451 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385855    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 13:55:31.394451 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385858    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 13:55:31.394451 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385860    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 13:55:31.394451 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385863    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 13:55:31.394451 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385867    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 13:55:31.395275 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385870    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 13:55:31.395275 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385873    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 13:55:31.395275 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385875    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 13:55:31.395275 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385879    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 13:55:31.395275 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385883    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 13:55:31.395275 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385886    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 13:55:31.395275 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385889    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 13:55:31.395275 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385892    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 13:55:31.395275 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385896    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 13:55:31.395275 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385899    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 13:55:31.395275 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385902    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 13:55:31.395275 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385905    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 13:55:31.395275 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385908    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 13:55:31.395275 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385911    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 13:55:31.395275 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385914    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 13:55:31.395275 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385917    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 13:55:31.395275 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385919    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 13:55:31.395275 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385922    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 13:55:31.395275 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385925    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 13:55:31.395275 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385928    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 13:55:31.396122 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385930    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 13:55:31.396122 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385933    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 13:55:31.396122 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385936    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 13:55:31.396122 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385938    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 13:55:31.396122 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385941    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 13:55:31.396122 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385943    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 13:55:31.396122 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385946    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 13:55:31.396122 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385949    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 13:55:31.396122 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385951    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 13:55:31.396122 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385954    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 13:55:31.396122 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385956    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 13:55:31.396122 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385959    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 13:55:31.396122 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385962    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 13:55:31.396122 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385965    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 13:55:31.396122 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385969    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 13:55:31.396122 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385972    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 13:55:31.396122 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385975    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 13:55:31.396122 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385978    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 13:55:31.396122 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385981    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 13:55:31.396122 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385984    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 13:55:31.397045 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385988    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 13:55:31.397045 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385991    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 13:55:31.397045 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385993    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 13:55:31.397045 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385996    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 13:55:31.397045 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.385999    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 13:55:31.397045 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.386001    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 13:55:31.397045 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.386004    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 13:55:31.397045 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.386006    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 13:55:31.397045 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.386009    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 13:55:31.397045 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.386011    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 13:55:31.397045 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.386014    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 13:55:31.397045 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.386016    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 13:55:31.397045 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.386019    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 13:55:31.397045 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.386022    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 13:55:31.397045 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.386025    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 13:55:31.397045 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.386027    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 13:55:31.397045 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.386029    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 13:55:31.397045 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.386032    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 13:55:31.397045 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.386034    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 13:55:31.397544 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.386037    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 13:55:31.397544 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.386039    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 13:55:31.397544 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.386042    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 13:55:31.397544 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.386044    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 13:55:31.397544 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.386047    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 13:55:31.397544 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.386049    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 13:55:31.397544 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.386051    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 13:55:31.397544 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.386054    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 13:55:31.397544 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.386056    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 13:55:31.397544 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.386059    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 13:55:31.397544 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.386062    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 13:55:31.397544 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.386064    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 13:55:31.397544 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.386067    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 13:55:31.397544 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.386070    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 13:55:31.397544 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.386073    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 13:55:31.397544 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.386075    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 13:55:31.397544 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.386078    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 13:55:31.397544 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.386080    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 13:55:31.397544 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.386083    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 13:55:31.397544 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.386085    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 13:55:31.398318 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.386088    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 13:55:31.398318 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.386091    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 13:55:31.398318 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.386093    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 13:55:31.398318 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.386096    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 13:55:31.398318 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.386098    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 13:55:31.398318 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.386833    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 13:55:31.398318 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.395395    2560 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 13:55:31.398318 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.395413    2560 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 13:55:31.398318 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.395435    2560 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 13:55:31.398318 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.395444    2560 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 13:55:31.398318 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395551    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 13:55:31.398318 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395559    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 13:55:31.398318 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395564    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 13:55:31.398318 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395569    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 13:55:31.398911 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395574    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 13:55:31.398911 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395579    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 13:55:31.398911 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395585    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 13:55:31.398911 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395592    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 13:55:31.398911 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395596    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 13:55:31.398911 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395601    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 13:55:31.398911 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395606    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 13:55:31.398911 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395610    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 13:55:31.398911 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395615    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 13:55:31.398911 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395628    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 13:55:31.398911 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395633    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 13:55:31.398911 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395637    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 13:55:31.398911 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395641    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 13:55:31.398911 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395646    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 13:55:31.398911 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395650    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 13:55:31.398911 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395654    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 13:55:31.398911 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395658    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 13:55:31.398911 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395662    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 13:55:31.398911 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395666    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 13:55:31.398911 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395670    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 13:55:31.399424 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395675    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 13:55:31.399424 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395679    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 13:55:31.399424 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395683    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 13:55:31.399424 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395688    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 13:55:31.399424 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395692    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 13:55:31.399424 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395697    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 13:55:31.399424 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395701    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 13:55:31.399424 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395708    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 13:55:31.399424 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395714    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 13:55:31.399424 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395718    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 13:55:31.399424 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395722    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 13:55:31.399424 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395726    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 13:55:31.399424 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395731    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 13:55:31.399424 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395735    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 13:55:31.399424 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395739    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 13:55:31.399424 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395743    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 13:55:31.399424 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395747    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 13:55:31.399424 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395751    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 13:55:31.399906 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395755    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 13:55:31.399906 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395759    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 13:55:31.399906 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395763    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 13:55:31.399906 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395767    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 13:55:31.399906 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395779    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 13:55:31.399906 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395783    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 13:55:31.399906 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395787    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 13:55:31.399906 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395792    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 13:55:31.399906 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395796    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 13:55:31.399906 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395800    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 13:55:31.399906 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395804    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 13:55:31.399906 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395809    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 13:55:31.399906 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395813    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 13:55:31.399906 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395817    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 13:55:31.399906 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395821    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 13:55:31.399906 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395825    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 13:55:31.399906 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395829    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 13:55:31.399906 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395833    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 13:55:31.399906 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395838    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 13:55:31.399906 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395842    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 13:55:31.400407 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395846    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 13:55:31.400407 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395850    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 13:55:31.400407 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395854    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 13:55:31.400407 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395858    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 13:55:31.400407 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395862    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 13:55:31.400407 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395866    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 13:55:31.400407 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395870    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 13:55:31.400407 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395874    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 13:55:31.400407 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395878    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 13:55:31.400407 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395882    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 13:55:31.400407 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395886    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 13:55:31.400407 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395890    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 13:55:31.400407 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395894    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 13:55:31.400407 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395898    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 13:55:31.400407 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395902    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 13:55:31.400407 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395905    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 13:55:31.400407 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395909    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 13:55:31.400407 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395923    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 13:55:31.400407 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395927    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 13:55:31.400407 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395931    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 13:55:31.401110 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395936    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 13:55:31.401110 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395941    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 13:55:31.401110 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395945    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 13:55:31.401110 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395949    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 13:55:31.401110 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395953    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 13:55:31.401110 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395958    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 13:55:31.401110 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395962    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 13:55:31.401110 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395966    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 13:55:31.401110 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395970    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 13:55:31.401110 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395975    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 13:55:31.401110 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395981    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 13:55:31.401110 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395986    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 13:55:31.401110 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395991    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 13:55:31.401110 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395995    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 13:55:31.401110 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.395999    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 13:55:31.401110 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396004    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 13:55:31.401110 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396007    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 13:55:31.401615 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.396016    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 13:55:31.401615 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396182    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 13:55:31.401615 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396191    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 13:55:31.401615 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396196    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 13:55:31.401615 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396200    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 13:55:31.401615 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396204    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 13:55:31.401615 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396209    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 13:55:31.401615 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396214    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 13:55:31.401615 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396218    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 13:55:31.401615 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396222    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 13:55:31.401615 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396226    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 13:55:31.401615 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396238    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 13:55:31.401615 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396242    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 13:55:31.401615 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396255    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 13:55:31.401615 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396260    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 13:55:31.402112 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396264    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 13:55:31.402112 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396287    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 13:55:31.402112 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396291    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 13:55:31.402112 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396296    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 13:55:31.402112 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396300    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 13:55:31.402112 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396304    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 13:55:31.402112 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396308    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 13:55:31.402112 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396313    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 13:55:31.402112 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396318    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 13:55:31.402112 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396322    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 13:55:31.402112 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396326    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 13:55:31.402112 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396331    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 13:55:31.402112 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396335    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 13:55:31.402112 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396339    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 13:55:31.402112 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396343    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 13:55:31.402112 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396348    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 13:55:31.402112 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396352    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 13:55:31.402112 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396356    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 13:55:31.402112 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396360    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 13:55:31.402112 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396364    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 13:55:31.402599 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396368    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 13:55:31.402599 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396372    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 13:55:31.402599 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396376    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 13:55:31.402599 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396380    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 13:55:31.402599 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396384    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 13:55:31.402599 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396388    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 13:55:31.402599 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396392    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 13:55:31.402599 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396396    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 13:55:31.402599 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396400    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 13:55:31.402599 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396404    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 13:55:31.402599 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396408    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 13:55:31.402599 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396421    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 13:55:31.402599 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396425    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 13:55:31.402599 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396429    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 13:55:31.402599 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396433    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 13:55:31.402599 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396437    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 13:55:31.402599 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396442    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 13:55:31.402599 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396446    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 13:55:31.402599 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396450    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 13:55:31.402599 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396453    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 13:55:31.403125 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396472    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 13:55:31.403125 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396476    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 13:55:31.403125 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396480    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 13:55:31.403125 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396484    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 13:55:31.403125 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396490    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 13:55:31.403125 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396496    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 13:55:31.403125 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396500    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 13:55:31.403125 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396504    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 13:55:31.403125 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396508    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 13:55:31.403125 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396511    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 13:55:31.403125 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396515    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 13:55:31.403125 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396519    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 13:55:31.403125 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396524    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 13:55:31.403125 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396529    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 13:55:31.403125 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396534    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 13:55:31.403125 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396537    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 13:55:31.403125 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396541    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 13:55:31.403125 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396545    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 13:55:31.403125 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396549    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 13:55:31.403602 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396553    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 13:55:31.403602 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396556    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 13:55:31.403602 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396560    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 13:55:31.403602 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396565    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 13:55:31.403602 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396569    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 13:55:31.403602 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396584    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 13:55:31.403602 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396589    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 13:55:31.403602 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396593    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 13:55:31.403602 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396598    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 13:55:31.403602 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396602    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 13:55:31.403602 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396606    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 13:55:31.403602 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396611    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 13:55:31.403602 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396614    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 13:55:31.403602 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396619    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 13:55:31.403602 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396622    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 13:55:31.403602 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396627    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 13:55:31.403602 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396631    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 13:55:31.403602 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396635    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 13:55:31.403602 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396639    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 13:55:31.404103 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396644    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 13:55:31.404103 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396648    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 13:55:31.404103 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396652    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 13:55:31.404103 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396658    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 13:55:31.404103 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396664    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 13:55:31.404103 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396669    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 13:55:31.404103 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:31.396673    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 13:55:31.404103 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.396682    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 13:55:31.404103 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.396841    2560 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 13:55:31.404103 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.401732    2560 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 13:55:31.404103 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.402839    2560 server.go:1019] "Starting client certificate rotation"
Sep 17 13:55:31.404103 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.402942    2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 13:55:31.404103 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.402975    2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 13:55:31.405759 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.405743    2560 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 13:55:31.410899 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.410877    2560 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 13:55:31.421066 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.421052    2560 log.go:25] "Validated CRI v1 runtime API"
Sep 17 13:55:31.424226 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.424212    2560 log.go:25] "Validated CRI v1 image API"
Sep 17 13:55:31.428039 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.428026    2560 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 13:55:31.432182 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.432165    2560 fs.go:135] Filesystem UUIDs: map[24c4f9dd-9443-49a2-8fbc-66d744c2dab3:/dev/nvme0n1p3 5b11dd88-e960-4c09-a902-ee93c3257ca1:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Sep 17 13:55:31.432259 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.432181    2560 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 13:55:31.438619 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.438507    2560 manager.go:216] Machine: {Timestamp:2026-09-17 13:55:31.436920068 +0000 UTC m=+0.379063709 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3200567 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2504e421fa4cdd086953d168563177 SystemUUID:ec2504e4-21fa-4cdd-0869-53d168563177 BootID:dbd3ec68-d6d0-46fa-931d-a494c2b310f3 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c1:fd:98:41 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c1:fd:98:41 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2a:91:75:ae:24:01 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 13:55:31.438619 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.438614    2560 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 13:55:31.438711 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.438681    2560 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 13:55:31.442084 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.442063    2560 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 13:55:31.442222 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.442089    2560 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-158.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 13:55:31.442987 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.442977    2560 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 13:55:31.443025 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.442996    2560 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 13:55:31.443025 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.443004    2560 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 13:55:31.443025 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.443015    2560 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 13:55:31.444292 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.444257    2560 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 13:55:31.444803 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.444788    2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 13:55:31.448788 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.448776    2560 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 13:55:31.448900 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.448891    2560 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 13:55:31.448938 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.448918    2560 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 13:55:31.448938 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.448933    2560 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 13:55:31.448991 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.448947    2560 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 13:55:31.449065 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.448999    2560 kubelet.go:398] "Adding apiserver pod source"
Sep 17 13:55:31.449065 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.449014    2560 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 13:55:31.449423 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.449411    2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 13:55:31.449456 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.449435    2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 13:55:31.452081 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.452026    2560 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 17 13:55:31.454868 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.454851    2560 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 13:55:31.454947 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.454881    2560 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 13:55:31.456906 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.456895    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 13:55:31.456955 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.456910    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 13:55:31.456955 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.456916    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 13:55:31.456955 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.456921    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 13:55:31.456955 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.456929    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 13:55:31.456955 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.456935    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 13:55:31.456955 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.456949    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 13:55:31.456955 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.456954    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 13:55:31.457132 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.456960    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 13:55:31.457132 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.456966    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 13:55:31.457132 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.456974    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 13:55:31.457132 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.456979    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 13:55:31.457950 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.457940    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 13:55:31.457950 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.457948    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 13:55:31.461664 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.461649    2560 server.go:1268] "Started kubelet"
Sep 17 13:55:31.461829 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.461734    2560 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 13:55:31.461891 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.461757    2560 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 13:55:31.461891 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.461867    2560 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 13:55:31.462587 ip-10-0-129-158 systemd[1]: Started Kubernetes Kubelet.
Sep 17 13:55:31.463522 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.462763    2560 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 13:55:31.463522 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.463256    2560 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 13:55:31.466279 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:31.466259    2560 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 13:55:31.466368 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.466355    2560 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-158.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 13:55:31.466416 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:31.466387    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-158.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 13:55:31.466416 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:31.466387    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 13:55:31.470313 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.470297    2560 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 13:55:31.470398 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.470312    2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 13:55:31.470398 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:31.470382    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-158.ec2.internal\" not found"
Sep 17 13:55:31.471135 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.471115    2560 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 13:55:31.471135 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.471137    2560 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 13:55:31.471269 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.471116    2560 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 13:55:31.471316 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.471288    2560 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 13:55:31.471316 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.471296    2560 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 13:55:31.471394 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.471324    2560 factory.go:55] Registering systemd factory
Sep 17 13:55:31.471394 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.471338    2560 factory.go:223] Registration of the systemd container factory successfully
Sep 17 13:55:31.471509 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:31.470238    2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-158.ec2.internal.18d6203914dd5913  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-158.ec2.internal,UID:ip-10-0-129-158.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-158.ec2.internal,},FirstTimestamp:2026-09-17 13:55:31.461626131 +0000 UTC m=+0.403769767,LastTimestamp:2026-09-17 13:55:31.461626131 +0000 UTC m=+0.403769767,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-158.ec2.internal,}"
Sep 17 13:55:31.471584 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.471575    2560 factory.go:153] Registering CRI-O factory
Sep 17 13:55:31.471630 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.471589    2560 factory.go:223] Registration of the crio container factory successfully
Sep 17 13:55:31.471670 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.471642    2560 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 13:55:31.471670 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.471660    2560 factory.go:103] Registering Raw factory
Sep 17 13:55:31.471749 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.471674    2560 manager.go:1196] Started watching for new ooms in manager
Sep 17 13:55:31.471981 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.471971    2560 manager.go:349] Starting recovery of all containers
Sep 17 13:55:31.479354 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:31.479179    2560 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-158.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 13:55:31.479691 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:31.479668    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 13:55:31.483329 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.483313    2560 manager.go:354] Recovery completed
Sep 17 13:55:31.489542 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.489531    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 13:55:31.491063 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.491037    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-158.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 13:55:31.491139 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.491074    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-158.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 13:55:31.491139 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.491084    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-158.ec2.internal" event="NodeHasSufficientPID"
Sep 17 13:55:31.491601 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.491584    2560 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 13:55:31.491601 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.491599    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 13:55:31.491713 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.491601    2560 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 13:55:31.491713 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.491709    2560 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 13:55:31.492356 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.492339    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-158.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 13:55:31.492433 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.492364    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-158.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 13:55:31.492433 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.492377    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-158.ec2.internal" event="NodeHasSufficientPID"
Sep 17 13:55:31.493137 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:31.493056    2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-158.ec2.internal.18d62039169e83b0  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-158.ec2.internal,UID:ip-10-0-129-158.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-158.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-158.ec2.internal,},FirstTimestamp:2026-09-17 13:55:31.491062704 +0000 UTC m=+0.433206339,LastTimestamp:2026-09-17 13:55:31.491062704 +0000 UTC m=+0.433206339,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-158.ec2.internal,}"
Sep 17 13:55:31.493941 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.493929    2560 policy_none.go:49] "None policy: Start"
Sep 17 13:55:31.493989 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.493945    2560 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 13:55:31.493989 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.493954    2560 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 13:55:31.495882 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.495872    2560 policy_none.go:47] "Start"
Sep 17 13:55:31.507948 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:31.507891    2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-158.ec2.internal.18d62039169ec3d9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-158.ec2.internal,UID:ip-10-0-129-158.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-129-158.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-129-158.ec2.internal,},FirstTimestamp:2026-09-17 13:55:31.491079129 +0000 UTC m=+0.433222764,LastTimestamp:2026-09-17 13:55:31.491079129 +0000 UTC m=+0.433222764,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-158.ec2.internal,}"
Sep 17 13:55:31.520570 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:31.520513    2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-158.ec2.internal.18d62039169ee79f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-158.ec2.internal,UID:ip-10-0-129-158.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-129-158.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-129-158.ec2.internal,},FirstTimestamp:2026-09-17 13:55:31.491088287 +0000 UTC m=+0.433231923,LastTimestamp:2026-09-17 13:55:31.491088287 +0000 UTC m=+0.433231923,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-158.ec2.internal,}"
Sep 17 13:55:31.543960 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.522996    2560 manager.go:337] "Starting Device Plugin manager"
Sep 17 13:55:31.543960 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:31.523052    2560 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 13:55:31.543960 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.523064    2560 server.go:87] "Starting device plugin registration server"
Sep 17 13:55:31.543960 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.523328    2560 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 13:55:31.543960 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.523340    2560 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 13:55:31.543960 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.523403    2560 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 13:55:31.543960 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.523488    2560 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 13:55:31.543960 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.523496    2560 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 13:55:31.543960 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:31.524084    2560 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 13:55:31.543960 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:31.524127    2560 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-158.ec2.internal\" not found"
Sep 17 13:55:31.543960 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:31.528947    2560 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-129-158.ec2.internal.18d62039169e83b0\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-158.ec2.internal.18d62039169e83b0  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-158.ec2.internal,UID:ip-10-0-129-158.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-158.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-158.ec2.internal,},FirstTimestamp:2026-09-17 13:55:31.491062704 +0000 UTC m=+0.433206339,LastTimestamp:2026-09-17 13:55:31.492354066 +0000 UTC m=+0.434497705,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-158.ec2.internal,}"
Sep 17 13:55:31.543960 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.533033    2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qdfz5"
Sep 17 13:55:31.544429 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.544319    2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qdfz5"
Sep 17 13:55:31.545157 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:31.545103    2560 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-129-158.ec2.internal.18d62039169ec3d9\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-158.ec2.internal.18d62039169ec3d9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-158.ec2.internal,UID:ip-10-0-129-158.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-129-158.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-129-158.ec2.internal,},FirstTimestamp:2026-09-17 13:55:31.491079129 +0000 UTC m=+0.433222764,LastTimestamp:2026-09-17 13:55:31.492369744 +0000 UTC m=+0.434513382,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-158.ec2.internal,}"
Sep 17 13:55:31.561332 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.561309    2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 13:55:31.562416 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.562401    2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 13:55:31.562517 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.562445    2560 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 13:55:31.562517 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.562479    2560 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 13:55:31.562622 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:31.562524    2560 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 13:55:31.564735 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.564717    2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 13:55:31.624414 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.624397    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 13:55:31.625199 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.625180    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-158.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 13:55:31.625274 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.625213    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-158.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 13:55:31.625274 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.625223    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-158.ec2.internal" event="NodeHasSufficientPID"
Sep 17 13:55:31.625274 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.625243    2560 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-158.ec2.internal"
Sep 17 13:55:31.634482 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.634394    2560 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-158.ec2.internal"
Sep 17 13:55:31.634482 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:31.634417    2560 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-158.ec2.internal\": node \"ip-10-0-129-158.ec2.internal\" not found"
Sep 17 13:55:31.653364 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:31.653346    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-158.ec2.internal\" not found"
Sep 17 13:55:31.663557 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.663544    2560 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-158.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-158.ec2.internal"]
Sep 17 13:55:31.663621 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.663612    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 13:55:31.664751 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.664732    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-158.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 13:55:31.664836 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.664756    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-158.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 13:55:31.664836 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.664766    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-158.ec2.internal" event="NodeHasSufficientPID"
Sep 17 13:55:31.667051 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.667039    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 13:55:31.667179 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.667166    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-158.ec2.internal"
Sep 17 13:55:31.667223 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.667195    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 13:55:31.667691 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.667661    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-158.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 13:55:31.667691 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.667684    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-158.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 13:55:31.667836 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.667697    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-158.ec2.internal" event="NodeHasSufficientPID"
Sep 17 13:55:31.667836 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.667703    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-158.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 13:55:31.667836 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.667722    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-158.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 13:55:31.667836 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.667732    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-158.ec2.internal" event="NodeHasSufficientPID"
Sep 17 13:55:31.669899 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.669883    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-158.ec2.internal"
Sep 17 13:55:31.669973 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.669909    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 13:55:31.670625 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.670608    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-158.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 13:55:31.670706 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.670642    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-158.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 13:55:31.670706 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.670656    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-158.ec2.internal" event="NodeHasSufficientPID"
Sep 17 13:55:31.672142 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.672123    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/201ffe21c4e43373b0c674db075b035e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-158.ec2.internal\" (UID: \"201ffe21c4e43373b0c674db075b035e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-158.ec2.internal"
Sep 17 13:55:31.672203 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.672155    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9a84bb5c5cbac6dbd226621c20c98385-config\") pod \"kube-apiserver-proxy-ip-10-0-129-158.ec2.internal\" (UID: \"9a84bb5c5cbac6dbd226621c20c98385\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-158.ec2.internal"
Sep 17 13:55:31.672203 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.672182    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/201ffe21c4e43373b0c674db075b035e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-158.ec2.internal\" (UID: \"201ffe21c4e43373b0c674db075b035e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-158.ec2.internal"
Sep 17 13:55:31.687981 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:31.687964    2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-158.ec2.internal\" not found" node="ip-10-0-129-158.ec2.internal"
Sep 17 13:55:31.692323 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:31.692310    2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-158.ec2.internal\" not found" node="ip-10-0-129-158.ec2.internal"
Sep 17 13:55:31.754304 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:31.754286    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-158.ec2.internal\" not found"
Sep 17 13:55:31.772946 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.772927    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9a84bb5c5cbac6dbd226621c20c98385-config\") pod \"kube-apiserver-proxy-ip-10-0-129-158.ec2.internal\" (UID: \"9a84bb5c5cbac6dbd226621c20c98385\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-158.ec2.internal"
Sep 17 13:55:31.772999 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.772950    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/201ffe21c4e43373b0c674db075b035e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-158.ec2.internal\" (UID: \"201ffe21c4e43373b0c674db075b035e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-158.ec2.internal"
Sep 17 13:55:31.772999 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.772965    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/201ffe21c4e43373b0c674db075b035e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-158.ec2.internal\" (UID: \"201ffe21c4e43373b0c674db075b035e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-158.ec2.internal"
Sep 17 13:55:31.773071 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.772998    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/201ffe21c4e43373b0c674db075b035e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-158.ec2.internal\" (UID: \"201ffe21c4e43373b0c674db075b035e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-158.ec2.internal"
Sep 17 13:55:31.773071 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.773030    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9a84bb5c5cbac6dbd226621c20c98385-config\") pod \"kube-apiserver-proxy-ip-10-0-129-158.ec2.internal\" (UID: \"9a84bb5c5cbac6dbd226621c20c98385\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-158.ec2.internal"
Sep 17 13:55:31.773071 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.773034    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/201ffe21c4e43373b0c674db075b035e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-158.ec2.internal\" (UID: \"201ffe21c4e43373b0c674db075b035e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-158.ec2.internal"
Sep 17 13:55:31.855012 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:31.854986    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-158.ec2.internal\" not found"
Sep 17 13:55:31.955764 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:31.955714    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-158.ec2.internal\" not found"
Sep 17 13:55:31.990213 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.990198    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-158.ec2.internal"
Sep 17 13:55:31.995369 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:31.995275    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-158.ec2.internal"
Sep 17 13:55:32.056829 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:32.056798    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-158.ec2.internal\" not found"
Sep 17 13:55:32.157324 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:32.157303    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-158.ec2.internal\" not found"
Sep 17 13:55:32.257785 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:32.257739    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-158.ec2.internal\" not found"
Sep 17 13:55:32.358594 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:32.358571    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-158.ec2.internal\" not found"
Sep 17 13:55:32.402639 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:32.402621    2560 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 13:55:32.402986 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:32.402754    2560 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 13:55:32.459499 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:32.459478    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-158.ec2.internal\" not found"
Sep 17 13:55:32.550846 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:32.550774    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 13:50:32 +0000 UTC" deadline="2028-03-19 04:28:14.841126514 +0000 UTC"
Sep 17 13:55:32.550846 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:32.550806    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13166h32m42.290323844s"
Sep 17 13:55:32.558719 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:32.558697    2560 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 13:55:32.560130 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:32.560109    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-158.ec2.internal\" not found"
Sep 17 13:55:32.582393 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:32.582358    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod201ffe21c4e43373b0c674db075b035e.slice/crio-ee7f27e330dc3401522787d7be5f2a20dc8856df43ad7a99ff86f0eb1b440a0f WatchSource:0}: Error finding container ee7f27e330dc3401522787d7be5f2a20dc8856df43ad7a99ff86f0eb1b440a0f: Status 404 returned error can't find the container with id ee7f27e330dc3401522787d7be5f2a20dc8856df43ad7a99ff86f0eb1b440a0f
Sep 17 13:55:32.586299 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:32.586275    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 13:55:32.586392 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:32.586378    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 13:55:32.588657 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:32.588572    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-158.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 13:55:32.588657 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:32.588599    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-158.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 13:55:32.588657 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:32.588615    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-158.ec2.internal" event="NodeHasSufficientPID"
Sep 17 13:55:32.661727 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:32.661689    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-158.ec2.internal\" not found"
Sep 17 13:55:32.763060 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:32.763026    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-158.ec2.internal\" not found"
Sep 17 13:55:32.793879 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:32.793846    2560 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 13:55:32.809135 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:32.809072    2560 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 13:55:32.870965 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:32.870938    2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-158.ec2.internal"
Sep 17 13:55:32.885392 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:32.885368    2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 13:55:32.886633 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:32.886618    2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-158.ec2.internal"
Sep 17 13:55:32.903576 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:32.903550    2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 13:55:33.343154 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.343106    2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 13:55:33.450752 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.450723    2560 apiserver.go:52] "Watching apiserver"
Sep 17 13:55:33.460272 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.460248    2560 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 13:55:33.462692 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.462667    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-6546h","kube-system/konnectivity-agent-gqsqk","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6gqm","openshift-cluster-node-tuning-operator/tuned-zxhg5","openshift-dns/node-resolver-2zqvg","openshift-image-registry/node-ca-9wpxw","openshift-multus/multus-additional-cni-plugins-ssnxx","kube-system/global-pull-secret-syncer-58g8x","kube-system/kube-apiserver-proxy-ip-10-0-129-158.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-158.ec2.internal","openshift-multus/multus-vqb5r","openshift-multus/network-metrics-daemon-dqcvb","openshift-network-diagnostics/network-check-target-9jwwx","openshift-network-operator/iptables-alerter-xrrk6"]
Sep 17 13:55:33.465444 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.465423    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ssnxx"
Sep 17 13:55:33.468432 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.467792    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9jwwx"
Sep 17 13:55:33.468432 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:33.467865    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9jwwx" podUID="8677b05f-e355-4c69-b16e-8e64d8b0e701"
Sep 17 13:55:33.468432 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.467980    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 13:55:33.468432 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.468231    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 13:55:33.468432 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.468375    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5p4g4\""
Sep 17 13:55:33.468754 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.468674    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 13:55:33.468754 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.468709    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 13:55:33.469006 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.468988    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 13:55:33.470747 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.470246    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6gqm"
Sep 17 13:55:33.471185 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.471166    2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 13:55:33.472852 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.472833    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zxhg5"
Sep 17 13:55:33.474100 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.474078    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 13:55:33.474320 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.474300    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 13:55:33.474591 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.474574    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 13:55:33.474797 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.474779    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-h5nkd\""
Sep 17 13:55:33.475347 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.475236    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2zqvg"
Sep 17 13:55:33.478290 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.477717    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 13:55:33.478290 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.477995    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8kz7q\""
Sep 17 13:55:33.478290 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.478148    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 13:55:33.478290 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.478188    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 13:55:33.478588 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.478363    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xrrk6"
Sep 17 13:55:33.478588 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.478481    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2qxlc\""
Sep 17 13:55:33.478802 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.478780    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 13:55:33.480360 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.480340    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqcvb"
Sep 17 13:55:33.480465 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:33.480443    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqcvb" podUID="d12b5b83-268b-404c-b77b-dd8ec7ea24d1"
Sep 17 13:55:33.481563 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.481544    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 13:55:33.481901 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.481882    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7jffz\""
Sep 17 13:55:33.482499 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.482467    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/714b66d5-d63d-4ac3-a097-f338deef9592-socket-dir\") pod \"aws-ebs-csi-driver-node-r6gqm\" (UID: \"714b66d5-d63d-4ac3-a097-f338deef9592\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6gqm"
Sep 17 13:55:33.482606 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.482501    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bfe7d6ff-377c-4449-8c3c-69163deda18c-tmp\") pod \"tuned-zxhg5\" (UID: \"bfe7d6ff-377c-4449-8c3c-69163deda18c\") " pod="openshift-cluster-node-tuning-operator/tuned-zxhg5"
Sep 17 13:55:33.482707 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.482614    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/714b66d5-d63d-4ac3-a097-f338deef9592-device-dir\") pod \"aws-ebs-csi-driver-node-r6gqm\" (UID: \"714b66d5-d63d-4ac3-a097-f338deef9592\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6gqm"
Sep 17 13:55:33.482707 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.482670    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/714b66d5-d63d-4ac3-a097-f338deef9592-sys-fs\") pod \"aws-ebs-csi-driver-node-r6gqm\" (UID: \"714b66d5-d63d-4ac3-a097-f338deef9592\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6gqm"
Sep 17 13:55:33.482707 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.482703    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bfe7d6ff-377c-4449-8c3c-69163deda18c-etc-kubernetes\") pod \"tuned-zxhg5\" (UID: \"bfe7d6ff-377c-4449-8c3c-69163deda18c\") " pod="openshift-cluster-node-tuning-operator/tuned-zxhg5"
Sep 17 13:55:33.482838 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.482730    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bfe7d6ff-377c-4449-8c3c-69163deda18c-etc-sysctl-d\") pod \"tuned-zxhg5\" (UID: \"bfe7d6ff-377c-4449-8c3c-69163deda18c\") " pod="openshift-cluster-node-tuning-operator/tuned-zxhg5"
Sep 17 13:55:33.482838 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.482776    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/714b66d5-d63d-4ac3-a097-f338deef9592-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r6gqm\" (UID: \"714b66d5-d63d-4ac3-a097-f338deef9592\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6gqm"
Sep 17 13:55:33.482838 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.482801    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/714b66d5-d63d-4ac3-a097-f338deef9592-etc-selinux\") pod \"aws-ebs-csi-driver-node-r6gqm\" (UID: \"714b66d5-d63d-4ac3-a097-f338deef9592\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6gqm"
Sep 17 13:55:33.482997 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.482833    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bfe7d6ff-377c-4449-8c3c-69163deda18c-etc-modprobe-d\") pod \"tuned-zxhg5\" (UID: \"bfe7d6ff-377c-4449-8c3c-69163deda18c\") " pod="openshift-cluster-node-tuning-operator/tuned-zxhg5"
Sep 17 13:55:33.482997 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.482861    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bfe7d6ff-377c-4449-8c3c-69163deda18c-etc-sysctl-conf\") pod \"tuned-zxhg5\" (UID: \"bfe7d6ff-377c-4449-8c3c-69163deda18c\") " pod="openshift-cluster-node-tuning-operator/tuned-zxhg5"
Sep 17 13:55:33.483109 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.483029    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 13:55:33.483313 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.483288    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e2ae1c9d-26b0-47e9-ae74-53904cc9f089-system-cni-dir\") pod \"multus-additional-cni-plugins-ssnxx\" (UID: \"e2ae1c9d-26b0-47e9-ae74-53904cc9f089\") " pod="openshift-multus/multus-additional-cni-plugins-ssnxx"
Sep 17 13:55:33.483313 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.483309    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 13:55:33.483434 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.483330    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6546h"
Sep 17 13:55:33.483736 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.483332    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e2ae1c9d-26b0-47e9-ae74-53904cc9f089-cni-binary-copy\") pod \"multus-additional-cni-plugins-ssnxx\" (UID: \"e2ae1c9d-26b0-47e9-ae74-53904cc9f089\") " pod="openshift-multus/multus-additional-cni-plugins-ssnxx"
Sep 17 13:55:33.483828 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.483774    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/714b66d5-d63d-4ac3-a097-f338deef9592-registration-dir\") pod \"aws-ebs-csi-driver-node-r6gqm\" (UID: \"714b66d5-d63d-4ac3-a097-f338deef9592\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6gqm"
Sep 17 13:55:33.483885 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.483831    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e2ae1c9d-26b0-47e9-ae74-53904cc9f089-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ssnxx\" (UID: \"e2ae1c9d-26b0-47e9-ae74-53904cc9f089\") " pod="openshift-multus/multus-additional-cni-plugins-ssnxx"
Sep 17 13:55:33.483942 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.483880    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bfe7d6ff-377c-4449-8c3c-69163deda18c-sys\") pod \"tuned-zxhg5\" (UID: \"bfe7d6ff-377c-4449-8c3c-69163deda18c\") " pod="openshift-cluster-node-tuning-operator/tuned-zxhg5"
Sep 17 13:55:33.483942 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.483917    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bfe7d6ff-377c-4449-8c3c-69163deda18c-lib-modules\") pod \"tuned-zxhg5\" (UID: \"bfe7d6ff-377c-4449-8c3c-69163deda18c\") " pod="openshift-cluster-node-tuning-operator/tuned-zxhg5"
Sep 17 13:55:33.484049 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.483940    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bfe7d6ff-377c-4449-8c3c-69163deda18c-var-lib-kubelet\") pod \"tuned-zxhg5\" (UID: \"bfe7d6ff-377c-4449-8c3c-69163deda18c\") " pod="openshift-cluster-node-tuning-operator/tuned-zxhg5"
Sep 17 13:55:33.484049 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.483984    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e2ae1c9d-26b0-47e9-ae74-53904cc9f089-cnibin\") pod \"multus-additional-cni-plugins-ssnxx\" (UID: \"e2ae1c9d-26b0-47e9-ae74-53904cc9f089\") " pod="openshift-multus/multus-additional-cni-plugins-ssnxx"
Sep 17 13:55:33.484049 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.484011    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rndqv\" (UniqueName: \"kubernetes.io/projected/8677b05f-e355-4c69-b16e-8e64d8b0e701-kube-api-access-rndqv\") pod \"network-check-target-9jwwx\" (UID: \"8677b05f-e355-4c69-b16e-8e64d8b0e701\") " pod="openshift-network-diagnostics/network-check-target-9jwwx"
Sep 17 13:55:33.484201 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.484063    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bfe7d6ff-377c-4449-8c3c-69163deda18c-etc-systemd\") pod \"tuned-zxhg5\" (UID: \"bfe7d6ff-377c-4449-8c3c-69163deda18c\") " pod="openshift-cluster-node-tuning-operator/tuned-zxhg5"
Sep 17 13:55:33.484201 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.484125    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bfe7d6ff-377c-4449-8c3c-69163deda18c-run\") pod \"tuned-zxhg5\" (UID: \"bfe7d6ff-377c-4449-8c3c-69163deda18c\") " pod="openshift-cluster-node-tuning-operator/tuned-zxhg5"
Sep 17 13:55:33.484201 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.484163    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr9df\" (UniqueName: \"kubernetes.io/projected/bfe7d6ff-377c-4449-8c3c-69163deda18c-kube-api-access-kr9df\") pod \"tuned-zxhg5\" (UID: \"bfe7d6ff-377c-4449-8c3c-69163deda18c\") " pod="openshift-cluster-node-tuning-operator/tuned-zxhg5"
Sep 17 13:55:33.484344 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.484215    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e2ae1c9d-26b0-47e9-ae74-53904cc9f089-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ssnxx\" (UID: \"e2ae1c9d-26b0-47e9-ae74-53904cc9f089\") " pod="openshift-multus/multus-additional-cni-plugins-ssnxx"
Sep 17 13:55:33.484344 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.484250    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e2ae1c9d-26b0-47e9-ae74-53904cc9f089-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ssnxx\" (UID: \"e2ae1c9d-26b0-47e9-ae74-53904cc9f089\") " pod="openshift-multus/multus-additional-cni-plugins-ssnxx"
Sep 17 13:55:33.484344 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.484280    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-556p4\" (UniqueName: \"kubernetes.io/projected/e2ae1c9d-26b0-47e9-ae74-53904cc9f089-kube-api-access-556p4\") pod \"multus-additional-cni-plugins-ssnxx\" (UID: \"e2ae1c9d-26b0-47e9-ae74-53904cc9f089\") " pod="openshift-multus/multus-additional-cni-plugins-ssnxx"
Sep 17 13:55:33.484344 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.484305    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npddg\" (UniqueName: \"kubernetes.io/projected/714b66d5-d63d-4ac3-a097-f338deef9592-kube-api-access-npddg\") pod \"aws-ebs-csi-driver-node-r6gqm\" (UID: \"714b66d5-d63d-4ac3-a097-f338deef9592\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6gqm"
Sep 17 13:55:33.484560 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.484364    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bfe7d6ff-377c-4449-8c3c-69163deda18c-etc-sysconfig\") pod \"tuned-zxhg5\" (UID: \"bfe7d6ff-377c-4449-8c3c-69163deda18c\") " pod="openshift-cluster-node-tuning-operator/tuned-zxhg5"
Sep 17 13:55:33.484560 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.484387    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bfe7d6ff-377c-4449-8c3c-69163deda18c-host\") pod \"tuned-zxhg5\" (UID: \"bfe7d6ff-377c-4449-8c3c-69163deda18c\") " pod="openshift-cluster-node-tuning-operator/tuned-zxhg5"
Sep 17 13:55:33.484560 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.484410    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bfe7d6ff-377c-4449-8c3c-69163deda18c-etc-tuned\") pod \"tuned-zxhg5\" (UID: \"bfe7d6ff-377c-4449-8c3c-69163deda18c\") " pod="openshift-cluster-node-tuning-operator/tuned-zxhg5"
Sep 17 13:55:33.486116 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.486096    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vqb5r"
Sep 17 13:55:33.487800 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.487751    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rvw22\""
Sep 17 13:55:33.488046 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.488027    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 13:55:33.488463 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.488443    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 13:55:33.489609 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.489075    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 13:55:33.489984 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.489964    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 13:55:33.490060 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.490000    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 13:55:33.490595 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.490548    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 13:55:33.491117 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.491092    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 13:55:33.491216 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.491118    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gqsqk"
Sep 17 13:55:33.491442 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.491420    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-cgk4l\""
Sep 17 13:55:33.493215 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.493192    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58g8x"
Sep 17 13:55:33.493313 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:33.493292    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-58g8x" podUID="14057e5c-c9fb-4894-869c-9780abc9c769"
Sep 17 13:55:33.493982 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.493942    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 13:55:33.494976 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.494956    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 13:55:33.495158 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.494962    2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 13:55:33.495377 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.495360    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-wkfzt\""
Sep 17 13:55:33.496005 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.495602    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9wpxw"
Sep 17 13:55:33.497868 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.497852    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4p85j\""
Sep 17 13:55:33.498003 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.497973    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 13:55:33.498174 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.498148    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 13:55:33.498247 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.498173    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 13:55:33.519265 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.519248    2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-gjdqp"
Sep 17 13:55:33.533113 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.533095    2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-gjdqp"
Sep 17 13:55:33.567567 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.567516    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-158.ec2.internal" event={"ID":"9a84bb5c5cbac6dbd226621c20c98385","Type":"ContainerStarted","Data":"a2724576b96607e583d8899d42786b99bb333518fe760b39fa15ddaeeacce5f8"}
Sep 17 13:55:33.569022 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.569001    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-158.ec2.internal" event={"ID":"201ffe21c4e43373b0c674db075b035e","Type":"ContainerStarted","Data":"ee7f27e330dc3401522787d7be5f2a20dc8856df43ad7a99ff86f0eb1b440a0f"}
Sep 17 13:55:33.572994 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.572969    2560 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 13:55:33.584669 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.584646    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e2ae1c9d-26b0-47e9-ae74-53904cc9f089-cnibin\") pod \"multus-additional-cni-plugins-ssnxx\" (UID: \"e2ae1c9d-26b0-47e9-ae74-53904cc9f089\") " pod="openshift-multus/multus-additional-cni-plugins-ssnxx"
Sep 17 13:55:33.584751 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.584684    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rndqv\" (UniqueName: \"kubernetes.io/projected/8677b05f-e355-4c69-b16e-8e64d8b0e701-kube-api-access-rndqv\") pod \"network-check-target-9jwwx\" (UID: \"8677b05f-e355-4c69-b16e-8e64d8b0e701\") " pod="openshift-network-diagnostics/network-check-target-9jwwx"
Sep 17 13:55:33.584751 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.584715    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/08545f35-761a-4929-9192-254ca6b76569-systemd-units\") pod \"ovnkube-node-6546h\" (UID: \"08545f35-761a-4929-9192-254ca6b76569\") " pod="openshift-ovn-kubernetes/ovnkube-node-6546h"
Sep 17 13:55:33.584751 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.584740    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08545f35-761a-4929-9192-254ca6b76569-var-lib-openvswitch\") pod \"ovnkube-node-6546h\" (UID: \"08545f35-761a-4929-9192-254ca6b76569\") " pod="openshift-ovn-kubernetes/ovnkube-node-6546h"
Sep 17 13:55:33.584925 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.584766    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bfe7d6ff-377c-4449-8c3c-69163deda18c-etc-systemd\") pod \"tuned-zxhg5\" (UID: \"bfe7d6ff-377c-4449-8c3c-69163deda18c\") " pod="openshift-cluster-node-tuning-operator/tuned-zxhg5"
Sep 17 13:55:33.584925 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.584780    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e2ae1c9d-26b0-47e9-ae74-53904cc9f089-cnibin\") pod \"multus-additional-cni-plugins-ssnxx\" (UID: \"e2ae1c9d-26b0-47e9-ae74-53904cc9f089\") " pod="openshift-multus/multus-additional-cni-plugins-ssnxx"
Sep 17 13:55:33.584925 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.584792    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bfe7d6ff-377c-4449-8c3c-69163deda18c-run\") pod \"tuned-zxhg5\" (UID: \"bfe7d6ff-377c-4449-8c3c-69163deda18c\") " pod="openshift-cluster-node-tuning-operator/tuned-zxhg5"
Sep 17 13:55:33.584925 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.584832    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08545f35-761a-4929-9192-254ca6b76569-etc-openvswitch\") pod \"ovnkube-node-6546h\" (UID: \"08545f35-761a-4929-9192-254ca6b76569\") " pod="openshift-ovn-kubernetes/ovnkube-node-6546h"
Sep 17 13:55:33.584925 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.584889    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bfe7d6ff-377c-4449-8c3c-69163deda18c-run\") pod \"tuned-zxhg5\" (UID: \"bfe7d6ff-377c-4449-8c3c-69163deda18c\") " pod="openshift-cluster-node-tuning-operator/tuned-zxhg5"
Sep 17 13:55:33.584925 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.584921    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/08545f35-761a-4929-9192-254ca6b76569-env-overrides\") pod \"ovnkube-node-6546h\" (UID: \"08545f35-761a-4929-9192-254ca6b76569\") " pod="openshift-ovn-kubernetes/ovnkube-node-6546h"
Sep 17 13:55:33.585192 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.584960    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npddg\" (UniqueName: \"kubernetes.io/projected/714b66d5-d63d-4ac3-a097-f338deef9592-kube-api-access-npddg\") pod \"aws-ebs-csi-driver-node-r6gqm\" (UID: \"714b66d5-d63d-4ac3-a097-f338deef9592\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6gqm"
Sep 17 13:55:33.585192 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.584961    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bfe7d6ff-377c-4449-8c3c-69163deda18c-etc-systemd\") pod \"tuned-zxhg5\" (UID: \"bfe7d6ff-377c-4449-8c3c-69163deda18c\") " pod="openshift-cluster-node-tuning-operator/tuned-zxhg5"
Sep 17 13:55:33.585192 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.585010    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bfe7d6ff-377c-4449-8c3c-69163deda18c-etc-sysconfig\") pod \"tuned-zxhg5\" (UID: \"bfe7d6ff-377c-4449-8c3c-69163deda18c\") " pod="openshift-cluster-node-tuning-operator/tuned-zxhg5"
Sep 17 13:55:33.585192 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.585138    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bfe7d6ff-377c-4449-8c3c-69163deda18c-etc-tuned\") pod \"tuned-zxhg5\" (UID: \"bfe7d6ff-377c-4449-8c3c-69163deda18c\") " pod="openshift-cluster-node-tuning-operator/tuned-zxhg5"
Sep 17 13:55:33.585192 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.585171    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bfe7d6ff-377c-4449-8c3c-69163deda18c-etc-sysconfig\") pod \"tuned-zxhg5\" (UID: \"bfe7d6ff-377c-4449-8c3c-69163deda18c\") " pod="openshift-cluster-node-tuning-operator/tuned-zxhg5"
Sep 17 13:55:33.585412 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.585209    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bfe7d6ff-377c-4449-8c3c-69163deda18c-tmp\") pod \"tuned-zxhg5\" (UID: \"bfe7d6ff-377c-4449-8c3c-69163deda18c\") " pod="openshift-cluster-node-tuning-operator/tuned-zxhg5"
Sep 17 13:55:33.585412 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.585229    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/08545f35-761a-4929-9192-254ca6b76569-host-run-netns\") pod \"ovnkube-node-6546h\" (UID: \"08545f35-761a-4929-9192-254ca6b76569\") " pod="openshift-ovn-kubernetes/ovnkube-node-6546h"
Sep 17 13:55:33.585412 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.585306    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/08545f35-761a-4929-9192-254ca6b76569-log-socket\") pod \"ovnkube-node-6546h\" (UID: \"08545f35-761a-4929-9192-254ca6b76569\") " pod="openshift-ovn-kubernetes/ovnkube-node-6546h"
Sep 17 13:55:33.585412 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.585349    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/eae24b6c-3322-4165-96fa-cd61dab923ca-konnectivity-ca\") pod \"konnectivity-agent-gqsqk\" (UID: \"eae24b6c-3322-4165-96fa-cd61dab923ca\") " pod="kube-system/konnectivity-agent-gqsqk"
Sep 17 13:55:33.585412 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.585378    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/714b66d5-d63d-4ac3-a097-f338deef9592-socket-dir\") pod \"aws-ebs-csi-driver-node-r6gqm\" (UID: \"714b66d5-d63d-4ac3-a097-f338deef9592\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6gqm"
Sep 17 13:55:33.585722 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.585416    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/08545f35-761a-4929-9192-254ca6b76569-host-cni-bin\") pod \"ovnkube-node-6546h\" (UID: \"08545f35-761a-4929-9192-254ca6b76569\") " pod="openshift-ovn-kubernetes/ovnkube-node-6546h"
Sep 17 13:55:33.585722 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.585443    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/08545f35-761a-4929-9192-254ca6b76569-ovn-node-metrics-cert\") pod \"ovnkube-node-6546h\" (UID: \"08545f35-761a-4929-9192-254ca6b76569\") " pod="openshift-ovn-kubernetes/ovnkube-node-6546h"
Sep 17 13:55:33.585722 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.585467    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ece625b2-eada-4304-a7e0-c96a322f289b-host-var-lib-cni-multus\") pod \"multus-vqb5r\" (UID: \"ece625b2-eada-4304-a7e0-c96a322f289b\") " pod="openshift-multus/multus-vqb5r"
Sep 17 13:55:33.585722 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.585513    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/14057e5c-c9fb-4894-869c-9780abc9c769-kubelet-config\") pod \"global-pull-secret-syncer-58g8x\" (UID: \"14057e5c-c9fb-4894-869c-9780abc9c769\") " pod="kube-system/global-pull-secret-syncer-58g8x"
Sep 17 13:55:33.585722 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.585538    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a4a8010a-d12b-4b1a-a0db-e1c64109b151-hosts-file\") pod \"node-resolver-2zqvg\" (UID: \"a4a8010a-d12b-4b1a-a0db-e1c64109b151\") " pod="openshift-dns/node-resolver-2zqvg"
Sep 17 13:55:33.585722 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.585572    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/714b66d5-d63d-4ac3-a097-f338deef9592-sys-fs\") pod \"aws-ebs-csi-driver-node-r6gqm\" (UID: \"714b66d5-d63d-4ac3-a097-f338deef9592\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6gqm"
Sep 17 13:55:33.585722 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.585588    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/714b66d5-d63d-4ac3-a097-f338deef9592-socket-dir\") pod \"aws-ebs-csi-driver-node-r6gqm\" (UID: \"714b66d5-d63d-4ac3-a097-f338deef9592\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6gqm"
Sep 17 13:55:33.585722 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.585598    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bfe7d6ff-377c-4449-8c3c-69163deda18c-etc-kubernetes\") pod \"tuned-zxhg5\" (UID: \"bfe7d6ff-377c-4449-8c3c-69163deda18c\") " pod="openshift-cluster-node-tuning-operator/tuned-zxhg5"
Sep 17 13:55:33.585722 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.585623    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bfe7d6ff-377c-4449-8c3c-69163deda18c-etc-sysctl-d\") pod \"tuned-zxhg5\" (UID: \"bfe7d6ff-377c-4449-8c3c-69163deda18c\") " pod="openshift-cluster-node-tuning-operator/tuned-zxhg5"
Sep 17 13:55:33.585722 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.585651    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ece625b2-eada-4304-a7e0-c96a322f289b-host-var-lib-cni-bin\") pod \"multus-vqb5r\" (UID: \"ece625b2-eada-4304-a7e0-c96a322f289b\") " pod="openshift-multus/multus-vqb5r"
Sep 17 13:55:33.585722 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.585703    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l695d\" (UniqueName: \"kubernetes.io/projected/4071dbb7-7923-4beb-856a-f03b5c7cfb98-kube-api-access-l695d\") pod \"iptables-alerter-xrrk6\" (UID: \"4071dbb7-7923-4beb-856a-f03b5c7cfb98\") " pod="openshift-network-operator/iptables-alerter-xrrk6"
Sep 17 13:55:33.586302 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.585737    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/714b66d5-d63d-4ac3-a097-f338deef9592-etc-selinux\") pod \"aws-ebs-csi-driver-node-r6gqm\" (UID: \"714b66d5-d63d-4ac3-a097-f338deef9592\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6gqm"
Sep 17 13:55:33.586302 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.585709    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/714b66d5-d63d-4ac3-a097-f338deef9592-sys-fs\") pod \"aws-ebs-csi-driver-node-r6gqm\" (UID: \"714b66d5-d63d-4ac3-a097-f338deef9592\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6gqm"
Sep 17 13:55:33.586302 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.585778    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bfe7d6ff-377c-4449-8c3c-69163deda18c-etc-modprobe-d\") pod \"tuned-zxhg5\" (UID: \"bfe7d6ff-377c-4449-8c3c-69163deda18c\") " pod="openshift-cluster-node-tuning-operator/tuned-zxhg5"
Sep 17 13:55:33.586302 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.585809    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ece625b2-eada-4304-a7e0-c96a322f289b-multus-conf-dir\") pod \"multus-vqb5r\" (UID: \"ece625b2-eada-4304-a7e0-c96a322f289b\") " pod="openshift-multus/multus-vqb5r"
Sep 17 13:55:33.586302 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.585813    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bfe7d6ff-377c-4449-8c3c-69163deda18c-etc-kubernetes\") pod \"tuned-zxhg5\" (UID: \"bfe7d6ff-377c-4449-8c3c-69163deda18c\") " pod="openshift-cluster-node-tuning-operator/tuned-zxhg5"
Sep 17 13:55:33.586302 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.585835    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e2ae1c9d-26b0-47e9-ae74-53904cc9f089-cni-binary-copy\") pod \"multus-additional-cni-plugins-ssnxx\" (UID: \"e2ae1c9d-26b0-47e9-ae74-53904cc9f089\") " pod="openshift-multus/multus-additional-cni-plugins-ssnxx"
Sep 17 13:55:33.586302 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.585863    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/08545f35-761a-4929-9192-254ca6b76569-host-cni-netd\") pod \"ovnkube-node-6546h\" (UID: \"08545f35-761a-4929-9192-254ca6b76569\") " pod="openshift-ovn-kubernetes/ovnkube-node-6546h"
Sep 17 13:55:33.586302 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.585903    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/714b66d5-d63d-4ac3-a097-f338deef9592-etc-selinux\") pod \"aws-ebs-csi-driver-node-r6gqm\" (UID: \"714b66d5-d63d-4ac3-a097-f338deef9592\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6gqm"
Sep 17 13:55:33.586302 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.585908    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08545f35-761a-4929-9192-254ca6b76569-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6546h\" (UID: \"08545f35-761a-4929-9192-254ca6b76569\") " pod="openshift-ovn-kubernetes/ovnkube-node-6546h"
Sep 17 13:55:33.586302 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.585938    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gtf9\" (UniqueName: \"kubernetes.io/projected/08545f35-761a-4929-9192-254ca6b76569-kube-api-access-4gtf9\") pod \"ovnkube-node-6546h\" (UID: \"08545f35-761a-4929-9192-254ca6b76569\") " pod="openshift-ovn-kubernetes/ovnkube-node-6546h"
Sep 17 13:55:33.586302 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.585964    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ece625b2-eada-4304-a7e0-c96a322f289b-cni-binary-copy\") pod \"multus-vqb5r\" (UID: \"ece625b2-eada-4304-a7e0-c96a322f289b\") " pod="openshift-multus/multus-vqb5r"
Sep 17 13:55:33.586302 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.585989    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ece625b2-eada-4304-a7e0-c96a322f289b-multus-daemon-config\") pod \"multus-vqb5r\" (UID: \"ece625b2-eada-4304-a7e0-c96a322f289b\") " pod="openshift-multus/multus-vqb5r"
Sep 17 13:55:33.586302 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.585997    2560 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 13:55:33.586302 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.585992    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bfe7d6ff-377c-4449-8c3c-69163deda18c-etc-sysctl-d\") pod \"tuned-zxhg5\" (UID: \"bfe7d6ff-377c-4449-8c3c-69163deda18c\") " pod="openshift-cluster-node-tuning-operator/tuned-zxhg5"
Sep 17 13:55:33.586302 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.586084    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bfe7d6ff-377c-4449-8c3c-69163deda18c-etc-modprobe-d\") pod \"tuned-zxhg5\" (UID: \"bfe7d6ff-377c-4449-8c3c-69163deda18c\") " pod="openshift-cluster-node-tuning-operator/tuned-zxhg5"
Sep 17 13:55:33.586302 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.586127    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bfe7d6ff-377c-4449-8c3c-69163deda18c-sys\") pod \"tuned-zxhg5\" (UID: \"bfe7d6ff-377c-4449-8c3c-69163deda18c\") " pod="openshift-cluster-node-tuning-operator/tuned-zxhg5"
Sep 17 13:55:33.586302 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.586155    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bfe7d6ff-377c-4449-8c3c-69163deda18c-lib-modules\") pod \"tuned-zxhg5\" (UID: \"bfe7d6ff-377c-4449-8c3c-69163deda18c\") " pod="openshift-cluster-node-tuning-operator/tuned-zxhg5"
Sep 17 13:55:33.587242 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.586191    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nw7k\" (UniqueName: \"kubernetes.io/projected/d12b5b83-268b-404c-b77b-dd8ec7ea24d1-kube-api-access-9nw7k\") pod \"network-metrics-daemon-dqcvb\" (UID: \"d12b5b83-268b-404c-b77b-dd8ec7ea24d1\") " pod="openshift-multus/network-metrics-daemon-dqcvb"
Sep 17 13:55:33.587242 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.586221    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ece625b2-eada-4304-a7e0-c96a322f289b-host-run-netns\") pod \"multus-vqb5r\" (UID: \"ece625b2-eada-4304-a7e0-c96a322f289b\") " pod="openshift-multus/multus-vqb5r"
Sep 17 13:55:33.587242 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.586347    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bfe7d6ff-377c-4449-8c3c-69163deda18c-lib-modules\") pod \"tuned-zxhg5\" (UID: \"bfe7d6ff-377c-4449-8c3c-69163deda18c\") " pod="openshift-cluster-node-tuning-operator/tuned-zxhg5"
Sep 17 13:55:33.587242 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.586440    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ece625b2-eada-4304-a7e0-c96a322f289b-host-run-multus-certs\") pod \"multus-vqb5r\" (UID: \"ece625b2-eada-4304-a7e0-c96a322f289b\") " pod="openshift-multus/multus-vqb5r"
Sep 17 13:55:33.587242 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.586472    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/08545f35-761a-4929-9192-254ca6b76569-run-ovn\") pod \"ovnkube-node-6546h\" (UID: \"08545f35-761a-4929-9192-254ca6b76569\") " pod="openshift-ovn-kubernetes/ovnkube-node-6546h"
Sep 17 13:55:33.587242 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.586481    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bfe7d6ff-377c-4449-8c3c-69163deda18c-sys\") pod \"tuned-zxhg5\" (UID: \"bfe7d6ff-377c-4449-8c3c-69163deda18c\") " pod="openshift-cluster-node-tuning-operator/tuned-zxhg5"
Sep 17 13:55:33.587242 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.586508    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08545f35-761a-4929-9192-254ca6b76569-host-run-ovn-kubernetes\") pod \"ovnkube-node-6546h\" (UID: \"08545f35-761a-4929-9192-254ca6b76569\") " pod="openshift-ovn-kubernetes/ovnkube-node-6546h"
Sep 17 13:55:33.587242 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.586561    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr9df\" (UniqueName: \"kubernetes.io/projected/bfe7d6ff-377c-4449-8c3c-69163deda18c-kube-api-access-kr9df\") pod \"tuned-zxhg5\" (UID: \"bfe7d6ff-377c-4449-8c3c-69163deda18c\") " pod="openshift-cluster-node-tuning-operator/tuned-zxhg5"
Sep 17 13:55:33.587242 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.586573    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e2ae1c9d-26b0-47e9-ae74-53904cc9f089-cni-binary-copy\") pod \"multus-additional-cni-plugins-ssnxx\" (UID: \"e2ae1c9d-26b0-47e9-ae74-53904cc9f089\") " pod="openshift-multus/multus-additional-cni-plugins-ssnxx"
Sep 17 13:55:33.587242 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.586603    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/08545f35-761a-4929-9192-254ca6b76569-host-kubelet\") pod \"ovnkube-node-6546h\" (UID: \"08545f35-761a-4929-9192-254ca6b76569\") " pod="openshift-ovn-kubernetes/ovnkube-node-6546h"
Sep 17 13:55:33.587242 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.586665    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/08545f35-761a-4929-9192-254ca6b76569-node-log\") pod \"ovnkube-node-6546h\" (UID: \"08545f35-761a-4929-9192-254ca6b76569\") " pod="openshift-ovn-kubernetes/ovnkube-node-6546h"
Sep 17 13:55:33.587242 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.586704    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/14057e5c-c9fb-4894-869c-9780abc9c769-dbus\") pod \"global-pull-secret-syncer-58g8x\" (UID: \"14057e5c-c9fb-4894-869c-9780abc9c769\") " pod="kube-system/global-pull-secret-syncer-58g8x"
Sep 17 13:55:33.587242 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.586755    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e2ae1c9d-26b0-47e9-ae74-53904cc9f089-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ssnxx\" (UID: \"e2ae1c9d-26b0-47e9-ae74-53904cc9f089\") " pod="openshift-multus/multus-additional-cni-plugins-ssnxx"
Sep 17 13:55:33.587242 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.586812    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e2ae1c9d-26b0-47e9-ae74-53904cc9f089-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ssnxx\" (UID: \"e2ae1c9d-26b0-47e9-ae74-53904cc9f089\") " pod="openshift-multus/multus-additional-cni-plugins-ssnxx"
Sep 17 13:55:33.587242 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.586854    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-556p4\" (UniqueName: \"kubernetes.io/projected/e2ae1c9d-26b0-47e9-ae74-53904cc9f089-kube-api-access-556p4\") pod \"multus-additional-cni-plugins-ssnxx\" (UID: \"e2ae1c9d-26b0-47e9-ae74-53904cc9f089\") " pod="openshift-multus/multus-additional-cni-plugins-ssnxx"
Sep 17 13:55:33.587242 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.586876    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bfe7d6ff-377c-4449-8c3c-69163deda18c-host\") pod \"tuned-zxhg5\" (UID: \"bfe7d6ff-377c-4449-8c3c-69163deda18c\") " pod="openshift-cluster-node-tuning-operator/tuned-zxhg5"
Sep 17 13:55:33.587242 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.586898    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ece625b2-eada-4304-a7e0-c96a322f289b-cnibin\") pod \"multus-vqb5r\" (UID: \"ece625b2-eada-4304-a7e0-c96a322f289b\") " pod="openshift-multus/multus-vqb5r"
Sep 17 13:55:33.588189 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.586966    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ece625b2-eada-4304-a7e0-c96a322f289b-host-var-lib-kubelet\") pod \"multus-vqb5r\" (UID: \"ece625b2-eada-4304-a7e0-c96a322f289b\") " pod="openshift-multus/multus-vqb5r"
Sep 17 13:55:33.588189 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.587005    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ece625b2-eada-4304-a7e0-c96a322f289b-etc-kubernetes\") pod \"multus-vqb5r\" (UID: \"ece625b2-eada-4304-a7e0-c96a322f289b\") " pod="openshift-multus/multus-vqb5r"
Sep 17 13:55:33.588189 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.587032    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-575ht\" (UniqueName: \"kubernetes.io/projected/ece625b2-eada-4304-a7e0-c96a322f289b-kube-api-access-575ht\") pod \"multus-vqb5r\" (UID: \"ece625b2-eada-4304-a7e0-c96a322f289b\") " pod="openshift-multus/multus-vqb5r"
Sep 17 13:55:33.588189 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.587055    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/08545f35-761a-4929-9192-254ca6b76569-run-systemd\") pod \"ovnkube-node-6546h\" (UID: \"08545f35-761a-4929-9192-254ca6b76569\") " pod="openshift-ovn-kubernetes/ovnkube-node-6546h"
Sep 17 13:55:33.588189 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.587071    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e2ae1c9d-26b0-47e9-ae74-53904cc9f089-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ssnxx\" (UID: \"e2ae1c9d-26b0-47e9-ae74-53904cc9f089\") " pod="openshift-multus/multus-additional-cni-plugins-ssnxx"
Sep 17 13:55:33.588189 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.587080    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ece625b2-eada-4304-a7e0-c96a322f289b-host-run-k8s-cni-cncf-io\") pod \"multus-vqb5r\" (UID: \"ece625b2-eada-4304-a7e0-c96a322f289b\") " pod="openshift-multus/multus-vqb5r"
Sep 17 13:55:33.588189 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.587164    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bfe7d6ff-377c-4449-8c3c-69163deda18c-host\") pod \"tuned-zxhg5\" (UID: \"bfe7d6ff-377c-4449-8c3c-69163deda18c\") " pod="openshift-cluster-node-tuning-operator/tuned-zxhg5"
Sep 17 13:55:33.588189 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.587334    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ece625b2-eada-4304-a7e0-c96a322f289b-hostroot\") pod \"multus-vqb5r\" (UID: \"ece625b2-eada-4304-a7e0-c96a322f289b\") " pod="openshift-multus/multus-vqb5r"
Sep 17 13:55:33.588189 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.587369    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/14057e5c-c9fb-4894-869c-9780abc9c769-original-pull-secret\") pod \"global-pull-secret-syncer-58g8x\" (UID: \"14057e5c-c9fb-4894-869c-9780abc9c769\") " pod="kube-system/global-pull-secret-syncer-58g8x"
Sep 17 13:55:33.588189 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.587396    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4071dbb7-7923-4beb-856a-f03b5c7cfb98-host-slash\") pod \"iptables-alerter-xrrk6\" (UID: \"4071dbb7-7923-4beb-856a-f03b5c7cfb98\") " pod="openshift-network-operator/iptables-alerter-xrrk6"
Sep 17 13:55:33.588189 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.587457    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/714b66d5-d63d-4ac3-a097-f338deef9592-device-dir\") pod \"aws-ebs-csi-driver-node-r6gqm\" (UID: \"714b66d5-d63d-4ac3-a097-f338deef9592\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6gqm"
Sep 17 13:55:33.588189 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.587521    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d12b5b83-268b-404c-b77b-dd8ec7ea24d1-metrics-certs\") pod \"network-metrics-daemon-dqcvb\" (UID: \"d12b5b83-268b-404c-b77b-dd8ec7ea24d1\") " pod="openshift-multus/network-metrics-daemon-dqcvb"
Sep 17 13:55:33.588189 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.587554    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/08545f35-761a-4929-9192-254ca6b76569-host-slash\") pod \"ovnkube-node-6546h\" (UID: \"08545f35-761a-4929-9192-254ca6b76569\") " pod="openshift-ovn-kubernetes/ovnkube-node-6546h"
Sep 17 13:55:33.588189 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.587557    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/714b66d5-d63d-4ac3-a097-f338deef9592-device-dir\") pod \"aws-ebs-csi-driver-node-r6gqm\" (UID: \"714b66d5-d63d-4ac3-a097-f338deef9592\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6gqm"
Sep 17 13:55:33.588189 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.587598    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/08545f35-761a-4929-9192-254ca6b76569-ovnkube-script-lib\") pod \"ovnkube-node-6546h\" (UID: \"08545f35-761a-4929-9192-254ca6b76569\") " pod="openshift-ovn-kubernetes/ovnkube-node-6546h"
Sep 17 13:55:33.588189 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.587624    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ece625b2-eada-4304-a7e0-c96a322f289b-system-cni-dir\") pod \"multus-vqb5r\" (UID: \"ece625b2-eada-4304-a7e0-c96a322f289b\") " pod="openshift-multus/multus-vqb5r"
Sep 17 13:55:33.588189 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.587654    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/714b66d5-d63d-4ac3-a097-f338deef9592-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r6gqm\" (UID: \"714b66d5-d63d-4ac3-a097-f338deef9592\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6gqm"
Sep 17 13:55:33.588924 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.587680    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bfe7d6ff-377c-4449-8c3c-69163deda18c-etc-sysctl-conf\") pod \"tuned-zxhg5\" (UID: \"bfe7d6ff-377c-4449-8c3c-69163deda18c\") " pod="openshift-cluster-node-tuning-operator/tuned-zxhg5"
Sep 17 13:55:33.588924 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.587716    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ece625b2-eada-4304-a7e0-c96a322f289b-multus-cni-dir\") pod \"multus-vqb5r\" (UID: \"ece625b2-eada-4304-a7e0-c96a322f289b\") " pod="openshift-multus/multus-vqb5r"
Sep 17 13:55:33.588924 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.587736    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/714b66d5-d63d-4ac3-a097-f338deef9592-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r6gqm\" (UID: \"714b66d5-d63d-4ac3-a097-f338deef9592\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6gqm"
Sep 17 13:55:33.588924 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.587763    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ece625b2-eada-4304-a7e0-c96a322f289b-multus-socket-dir-parent\") pod \"multus-vqb5r\" (UID: \"ece625b2-eada-4304-a7e0-c96a322f289b\") " pod="openshift-multus/multus-vqb5r"
Sep 17 13:55:33.588924 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.587832    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9fcb9893-714e-47b5-9b6f-37e706357ffe-serviceca\") pod \"node-ca-9wpxw\" (UID: \"9fcb9893-714e-47b5-9b6f-37e706357ffe\") " pod="openshift-image-registry/node-ca-9wpxw"
Sep 17 13:55:33.588924 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.587871    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bfe7d6ff-377c-4449-8c3c-69163deda18c-etc-sysctl-conf\") pod \"tuned-zxhg5\" (UID: \"bfe7d6ff-377c-4449-8c3c-69163deda18c\") " pod="openshift-cluster-node-tuning-operator/tuned-zxhg5"
Sep 17 13:55:33.588924 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.587926    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e2ae1c9d-26b0-47e9-ae74-53904cc9f089-system-cni-dir\") pod \"multus-additional-cni-plugins-ssnxx\" (UID: \"e2ae1c9d-26b0-47e9-ae74-53904cc9f089\") " pod="openshift-multus/multus-additional-cni-plugins-ssnxx"
Sep 17 13:55:33.588924 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.588001    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e2ae1c9d-26b0-47e9-ae74-53904cc9f089-system-cni-dir\") pod \"multus-additional-cni-plugins-ssnxx\" (UID: \"e2ae1c9d-26b0-47e9-ae74-53904cc9f089\") " pod="openshift-multus/multus-additional-cni-plugins-ssnxx"
Sep 17 13:55:33.588924 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.588028    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/714b66d5-d63d-4ac3-a097-f338deef9592-registration-dir\") pod \"aws-ebs-csi-driver-node-r6gqm\" (UID: \"714b66d5-d63d-4ac3-a097-f338deef9592\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6gqm"
Sep 17 13:55:33.588924 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.588099    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/714b66d5-d63d-4ac3-a097-f338deef9592-registration-dir\") pod \"aws-ebs-csi-driver-node-r6gqm\" (UID: \"714b66d5-d63d-4ac3-a097-f338deef9592\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6gqm"
Sep 17 13:55:33.588924 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.588211    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e2ae1c9d-26b0-47e9-ae74-53904cc9f089-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ssnxx\" (UID: \"e2ae1c9d-26b0-47e9-ae74-53904cc9f089\") " pod="openshift-multus/multus-additional-cni-plugins-ssnxx"
Sep 17 13:55:33.588924 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.588076    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08545f35-761a-4929-9192-254ca6b76569-run-openvswitch\") pod \"ovnkube-node-6546h\" (UID: \"08545f35-761a-4929-9192-254ca6b76569\") " pod="openshift-ovn-kubernetes/ovnkube-node-6546h"
Sep 17 13:55:33.588924 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.588279    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/08545f35-761a-4929-9192-254ca6b76569-ovnkube-config\") pod \"ovnkube-node-6546h\" (UID: \"08545f35-761a-4929-9192-254ca6b76569\") " pod="openshift-ovn-kubernetes/ovnkube-node-6546h"
Sep 17 13:55:33.588924 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.588325    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9fcb9893-714e-47b5-9b6f-37e706357ffe-host\") pod \"node-ca-9wpxw\" (UID: \"9fcb9893-714e-47b5-9b6f-37e706357ffe\") " pod="openshift-image-registry/node-ca-9wpxw"
Sep 17 13:55:33.588924 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.588350    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2tbt\" (UniqueName: \"kubernetes.io/projected/9fcb9893-714e-47b5-9b6f-37e706357ffe-kube-api-access-x2tbt\") pod \"node-ca-9wpxw\" (UID: \"9fcb9893-714e-47b5-9b6f-37e706357ffe\") " pod="openshift-image-registry/node-ca-9wpxw"
Sep 17 13:55:33.588924 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.588376    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8xxr\" (UniqueName: \"kubernetes.io/projected/a4a8010a-d12b-4b1a-a0db-e1c64109b151-kube-api-access-f8xxr\") pod \"node-resolver-2zqvg\" (UID: \"a4a8010a-d12b-4b1a-a0db-e1c64109b151\") " pod="openshift-dns/node-resolver-2zqvg"
Sep 17 13:55:33.589612 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.588419    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e2ae1c9d-26b0-47e9-ae74-53904cc9f089-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ssnxx\" (UID: \"e2ae1c9d-26b0-47e9-ae74-53904cc9f089\") " pod="openshift-multus/multus-additional-cni-plugins-ssnxx"
Sep 17 13:55:33.589612 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.588445    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bfe7d6ff-377c-4449-8c3c-69163deda18c-var-lib-kubelet\") pod \"tuned-zxhg5\" (UID: \"bfe7d6ff-377c-4449-8c3c-69163deda18c\") " pod="openshift-cluster-node-tuning-operator/tuned-zxhg5"
Sep 17 13:55:33.589612 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.588469    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/eae24b6c-3322-4165-96fa-cd61dab923ca-agent-certs\") pod \"konnectivity-agent-gqsqk\" (UID: \"eae24b6c-3322-4165-96fa-cd61dab923ca\") " pod="kube-system/konnectivity-agent-gqsqk"
Sep 17 13:55:33.589612 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.588493    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4a8010a-d12b-4b1a-a0db-e1c64109b151-tmp-dir\") pod \"node-resolver-2zqvg\" (UID: \"a4a8010a-d12b-4b1a-a0db-e1c64109b151\") " pod="openshift-dns/node-resolver-2zqvg"
Sep 17 13:55:33.589612 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.588540    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4071dbb7-7923-4beb-856a-f03b5c7cfb98-iptables-alerter-script\") pod \"iptables-alerter-xrrk6\" (UID: \"4071dbb7-7923-4beb-856a-f03b5c7cfb98\") " pod="openshift-network-operator/iptables-alerter-xrrk6"
Sep 17 13:55:33.589612 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.589102    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e2ae1c9d-26b0-47e9-ae74-53904cc9f089-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ssnxx\" (UID: \"e2ae1c9d-26b0-47e9-ae74-53904cc9f089\") " pod="openshift-multus/multus-additional-cni-plugins-ssnxx"
Sep 17 13:55:33.589612 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.589088    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bfe7d6ff-377c-4449-8c3c-69163deda18c-var-lib-kubelet\") pod \"tuned-zxhg5\" (UID: \"bfe7d6ff-377c-4449-8c3c-69163deda18c\") " pod="openshift-cluster-node-tuning-operator/tuned-zxhg5"
Sep 17 13:55:33.590530 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.590511    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bfe7d6ff-377c-4449-8c3c-69163deda18c-etc-tuned\") pod \"tuned-zxhg5\" (UID: \"bfe7d6ff-377c-4449-8c3c-69163deda18c\") " pod="openshift-cluster-node-tuning-operator/tuned-zxhg5"
Sep 17 13:55:33.590678 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.590659    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bfe7d6ff-377c-4449-8c3c-69163deda18c-tmp\") pod \"tuned-zxhg5\" (UID: \"bfe7d6ff-377c-4449-8c3c-69163deda18c\") " pod="openshift-cluster-node-tuning-operator/tuned-zxhg5"
Sep 17 13:55:33.600190 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:33.600123    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 13:55:33.600190 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:33.600150    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 13:55:33.600190 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:33.600164    2560 projected.go:196] Error preparing data for projected volume kube-api-access-rndqv for pod openshift-network-diagnostics/network-check-target-9jwwx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 13:55:33.600388 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:33.600241    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8677b05f-e355-4c69-b16e-8e64d8b0e701-kube-api-access-rndqv podName:8677b05f-e355-4c69-b16e-8e64d8b0e701 nodeName:}" failed. No retries permitted until 2026-09-17 13:55:34.100221138 +0000 UTC m=+3.042364790 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-rndqv" (UniqueName: "kubernetes.io/projected/8677b05f-e355-4c69-b16e-8e64d8b0e701-kube-api-access-rndqv") pod "network-check-target-9jwwx" (UID: "8677b05f-e355-4c69-b16e-8e64d8b0e701") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 13:55:33.600915 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.600889    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr9df\" (UniqueName: \"kubernetes.io/projected/bfe7d6ff-377c-4449-8c3c-69163deda18c-kube-api-access-kr9df\") pod \"tuned-zxhg5\" (UID: \"bfe7d6ff-377c-4449-8c3c-69163deda18c\") " pod="openshift-cluster-node-tuning-operator/tuned-zxhg5"
Sep 17 13:55:33.601928 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.601904    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npddg\" (UniqueName: \"kubernetes.io/projected/714b66d5-d63d-4ac3-a097-f338deef9592-kube-api-access-npddg\") pod \"aws-ebs-csi-driver-node-r6gqm\" (UID: \"714b66d5-d63d-4ac3-a097-f338deef9592\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6gqm"
Sep 17 13:55:33.603918 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.603868    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-556p4\" (UniqueName: \"kubernetes.io/projected/e2ae1c9d-26b0-47e9-ae74-53904cc9f089-kube-api-access-556p4\") pod \"multus-additional-cni-plugins-ssnxx\" (UID: \"e2ae1c9d-26b0-47e9-ae74-53904cc9f089\") " pod="openshift-multus/multus-additional-cni-plugins-ssnxx"
Sep 17 13:55:33.689040 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.689005    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08545f35-761a-4929-9192-254ca6b76569-var-lib-openvswitch\") pod \"ovnkube-node-6546h\" (UID: \"08545f35-761a-4929-9192-254ca6b76569\") " pod="openshift-ovn-kubernetes/ovnkube-node-6546h"
Sep 17 13:55:33.689040 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.689043    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08545f35-761a-4929-9192-254ca6b76569-etc-openvswitch\") pod \"ovnkube-node-6546h\" (UID: \"08545f35-761a-4929-9192-254ca6b76569\") " pod="openshift-ovn-kubernetes/ovnkube-node-6546h"
Sep 17 13:55:33.689293 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.689059    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/08545f35-761a-4929-9192-254ca6b76569-env-overrides\") pod \"ovnkube-node-6546h\" (UID: \"08545f35-761a-4929-9192-254ca6b76569\") " pod="openshift-ovn-kubernetes/ovnkube-node-6546h"
Sep 17 13:55:33.689293 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.689088    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/08545f35-761a-4929-9192-254ca6b76569-host-run-netns\") pod \"ovnkube-node-6546h\" (UID: \"08545f35-761a-4929-9192-254ca6b76569\") " pod="openshift-ovn-kubernetes/ovnkube-node-6546h"
Sep 17 13:55:33.689293 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.689107    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/08545f35-761a-4929-9192-254ca6b76569-log-socket\") pod \"ovnkube-node-6546h\" (UID: \"08545f35-761a-4929-9192-254ca6b76569\") " pod="openshift-ovn-kubernetes/ovnkube-node-6546h"
Sep 17 13:55:33.689293 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.689125    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/eae24b6c-3322-4165-96fa-cd61dab923ca-konnectivity-ca\") pod \"konnectivity-agent-gqsqk\" (UID: \"eae24b6c-3322-4165-96fa-cd61dab923ca\") " pod="kube-system/konnectivity-agent-gqsqk"
Sep 17 13:55:33.689293 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.689149    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/08545f35-761a-4929-9192-254ca6b76569-host-cni-bin\") pod \"ovnkube-node-6546h\" (UID: \"08545f35-761a-4929-9192-254ca6b76569\") " pod="openshift-ovn-kubernetes/ovnkube-node-6546h"
Sep 17 13:55:33.689293 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.689172    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/08545f35-761a-4929-9192-254ca6b76569-ovn-node-metrics-cert\") pod \"ovnkube-node-6546h\" (UID: \"08545f35-761a-4929-9192-254ca6b76569\") " pod="openshift-ovn-kubernetes/ovnkube-node-6546h"
Sep 17 13:55:33.689293 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.689200    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ece625b2-eada-4304-a7e0-c96a322f289b-host-var-lib-cni-multus\") pod \"multus-vqb5r\" (UID: \"ece625b2-eada-4304-a7e0-c96a322f289b\") " pod="openshift-multus/multus-vqb5r"
Sep 17 13:55:33.689293 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.689223    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/14057e5c-c9fb-4894-869c-9780abc9c769-kubelet-config\") pod \"global-pull-secret-syncer-58g8x\" (UID: \"14057e5c-c9fb-4894-869c-9780abc9c769\") " pod="kube-system/global-pull-secret-syncer-58g8x"
Sep 17 13:55:33.689293 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.689240    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a4a8010a-d12b-4b1a-a0db-e1c64109b151-hosts-file\") pod \"node-resolver-2zqvg\" (UID: \"a4a8010a-d12b-4b1a-a0db-e1c64109b151\") " pod="openshift-dns/node-resolver-2zqvg"
Sep 17 13:55:33.689293 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.689257    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ece625b2-eada-4304-a7e0-c96a322f289b-host-var-lib-cni-bin\") pod \"multus-vqb5r\" (UID: \"ece625b2-eada-4304-a7e0-c96a322f289b\") " pod="openshift-multus/multus-vqb5r"
Sep 17 13:55:33.689293 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.689275    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l695d\" (UniqueName: \"kubernetes.io/projected/4071dbb7-7923-4beb-856a-f03b5c7cfb98-kube-api-access-l695d\") pod \"iptables-alerter-xrrk6\" (UID: \"4071dbb7-7923-4beb-856a-f03b5c7cfb98\") " pod="openshift-network-operator/iptables-alerter-xrrk6"
Sep 17 13:55:33.689293 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.689301    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ece625b2-eada-4304-a7e0-c96a322f289b-multus-conf-dir\") pod \"multus-vqb5r\" (UID: \"ece625b2-eada-4304-a7e0-c96a322f289b\") " pod="openshift-multus/multus-vqb5r"
Sep 17 13:55:33.689929 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.689325    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/08545f35-761a-4929-9192-254ca6b76569-host-cni-netd\") pod \"ovnkube-node-6546h\" (UID: \"08545f35-761a-4929-9192-254ca6b76569\") " pod="openshift-ovn-kubernetes/ovnkube-node-6546h"
Sep 17 13:55:33.689929 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.689346    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08545f35-761a-4929-9192-254ca6b76569-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6546h\" (UID: \"08545f35-761a-4929-9192-254ca6b76569\") " pod="openshift-ovn-kubernetes/ovnkube-node-6546h"
Sep 17 13:55:33.689929 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.689366    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gtf9\" (UniqueName: \"kubernetes.io/projected/08545f35-761a-4929-9192-254ca6b76569-kube-api-access-4gtf9\") pod \"ovnkube-node-6546h\" (UID: \"08545f35-761a-4929-9192-254ca6b76569\") " pod="openshift-ovn-kubernetes/ovnkube-node-6546h"
Sep 17 13:55:33.689929 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.689387    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ece625b2-eada-4304-a7e0-c96a322f289b-cni-binary-copy\") pod \"multus-vqb5r\" (UID: \"ece625b2-eada-4304-a7e0-c96a322f289b\") " pod="openshift-multus/multus-vqb5r"
Sep 17 13:55:33.689929 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.689405    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ece625b2-eada-4304-a7e0-c96a322f289b-multus-daemon-config\") pod \"multus-vqb5r\" (UID: \"ece625b2-eada-4304-a7e0-c96a322f289b\") " pod="openshift-multus/multus-vqb5r"
Sep 17 13:55:33.689929 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.689434    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nw7k\" (UniqueName: \"kubernetes.io/projected/d12b5b83-268b-404c-b77b-dd8ec7ea24d1-kube-api-access-9nw7k\") pod \"network-metrics-daemon-dqcvb\" (UID: \"d12b5b83-268b-404c-b77b-dd8ec7ea24d1\") " pod="openshift-multus/network-metrics-daemon-dqcvb"
Sep 17 13:55:33.689929 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.689455    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ece625b2-eada-4304-a7e0-c96a322f289b-host-run-netns\") pod \"multus-vqb5r\" (UID: \"ece625b2-eada-4304-a7e0-c96a322f289b\") " pod="openshift-multus/multus-vqb5r"
Sep 17 13:55:33.689929 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.689471    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ece625b2-eada-4304-a7e0-c96a322f289b-host-run-multus-certs\") pod \"multus-vqb5r\" (UID: \"ece625b2-eada-4304-a7e0-c96a322f289b\") " pod="openshift-multus/multus-vqb5r"
Sep 17 13:55:33.689929 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.689487    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/08545f35-761a-4929-9192-254ca6b76569-run-ovn\") pod \"ovnkube-node-6546h\" (UID: \"08545f35-761a-4929-9192-254ca6b76569\") " pod="openshift-ovn-kubernetes/ovnkube-node-6546h"
Sep 17 13:55:33.689929 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.689502    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08545f35-761a-4929-9192-254ca6b76569-host-run-ovn-kubernetes\") pod \"ovnkube-node-6546h\" (UID: \"08545f35-761a-4929-9192-254ca6b76569\") " pod="openshift-ovn-kubernetes/ovnkube-node-6546h"
Sep 17 13:55:33.689929 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.689532    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/08545f35-761a-4929-9192-254ca6b76569-host-kubelet\") pod \"ovnkube-node-6546h\" (UID: \"08545f35-761a-4929-9192-254ca6b76569\") " pod="openshift-ovn-kubernetes/ovnkube-node-6546h"
Sep 17 13:55:33.689929 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.689559    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/08545f35-761a-4929-9192-254ca6b76569-node-log\") pod \"ovnkube-node-6546h\" (UID: \"08545f35-761a-4929-9192-254ca6b76569\") " pod="openshift-ovn-kubernetes/ovnkube-node-6546h"
Sep 17 13:55:33.689929 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.689575    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/14057e5c-c9fb-4894-869c-9780abc9c769-dbus\") pod \"global-pull-secret-syncer-58g8x\" (UID: \"14057e5c-c9fb-4894-869c-9780abc9c769\") " pod="kube-system/global-pull-secret-syncer-58g8x"
Sep 17 13:55:33.689929 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.689603    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ece625b2-eada-4304-a7e0-c96a322f289b-cnibin\") pod \"multus-vqb5r\" (UID: \"ece625b2-eada-4304-a7e0-c96a322f289b\") " pod="openshift-multus/multus-vqb5r"
Sep 17 13:55:33.689929 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.689629    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ece625b2-eada-4304-a7e0-c96a322f289b-host-var-lib-kubelet\") pod \"multus-vqb5r\" (UID: \"ece625b2-eada-4304-a7e0-c96a322f289b\") " pod="openshift-multus/multus-vqb5r"
Sep 17 13:55:33.689929 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.689653    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ece625b2-eada-4304-a7e0-c96a322f289b-etc-kubernetes\") pod \"multus-vqb5r\" (UID: \"ece625b2-eada-4304-a7e0-c96a322f289b\") " pod="openshift-multus/multus-vqb5r"
Sep 17 13:55:33.689929 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.689691    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ece625b2-eada-4304-a7e0-c96a322f289b-multus-conf-dir\") pod \"multus-vqb5r\" (UID: \"ece625b2-eada-4304-a7e0-c96a322f289b\") " pod="openshift-multus/multus-vqb5r"
Sep 17 13:55:33.689929 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.689719    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/08545f35-761a-4929-9192-254ca6b76569-host-cni-bin\") pod \"ovnkube-node-6546h\" (UID: \"08545f35-761a-4929-9192-254ca6b76569\") " pod="openshift-ovn-kubernetes/ovnkube-node-6546h"
Sep 17 13:55:33.690915 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.689728    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/08545f35-761a-4929-9192-254ca6b76569-host-cni-netd\") pod \"ovnkube-node-6546h\" (UID: \"08545f35-761a-4929-9192-254ca6b76569\") " pod="openshift-ovn-kubernetes/ovnkube-node-6546h"
Sep 17 13:55:33.690915 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.689724    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/08545f35-761a-4929-9192-254ca6b76569-host-kubelet\") pod \"ovnkube-node-6546h\" (UID: \"08545f35-761a-4929-9192-254ca6b76569\") " pod="openshift-ovn-kubernetes/ovnkube-node-6546h"
Sep 17 13:55:33.690915 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.689774    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08545f35-761a-4929-9192-254ca6b76569-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6546h\" (UID: \"08545f35-761a-4929-9192-254ca6b76569\") " pod="openshift-ovn-kubernetes/ovnkube-node-6546h"
Sep 17 13:55:33.690915 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.689781    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ece625b2-eada-4304-a7e0-c96a322f289b-host-run-netns\") pod \"multus-vqb5r\" (UID: \"ece625b2-eada-4304-a7e0-c96a322f289b\") " pod="openshift-multus/multus-vqb5r"
Sep 17 13:55:33.690915 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.689848    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/14057e5c-c9fb-4894-869c-9780abc9c769-dbus\") pod \"global-pull-secret-syncer-58g8x\" (UID: \"14057e5c-c9fb-4894-869c-9780abc9c769\") " pod="kube-system/global-pull-secret-syncer-58g8x"
Sep 17 13:55:33.690915 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.689900    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ece625b2-eada-4304-a7e0-c96a322f289b-cnibin\") pod \"multus-vqb5r\" (UID: \"ece625b2-eada-4304-a7e0-c96a322f289b\") " pod="openshift-multus/multus-vqb5r"
Sep 17 13:55:33.690915 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.689943    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ece625b2-eada-4304-a7e0-c96a322f289b-host-var-lib-kubelet\") pod \"multus-vqb5r\" (UID: \"ece625b2-eada-4304-a7e0-c96a322f289b\") " pod="openshift-multus/multus-vqb5r"
Sep 17 13:55:33.690915 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.689981    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/08545f35-761a-4929-9192-254ca6b76569-host-run-netns\") pod \"ovnkube-node-6546h\" (UID: \"08545f35-761a-4929-9192-254ca6b76569\") " pod="openshift-ovn-kubernetes/ovnkube-node-6546h"
Sep 17 13:55:33.690915 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.690001    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08545f35-761a-4929-9192-254ca6b76569-host-run-ovn-kubernetes\") pod \"ovnkube-node-6546h\" (UID: \"08545f35-761a-4929-9192-254ca6b76569\") " pod="openshift-ovn-kubernetes/ovnkube-node-6546h"
Sep 17 13:55:33.690915 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.690018    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08545f35-761a-4929-9192-254ca6b76569-var-lib-openvswitch\") pod \"ovnkube-node-6546h\" (UID: \"08545f35-761a-4929-9192-254ca6b76569\") " pod="openshift-ovn-kubernetes/ovnkube-node-6546h"
Sep 17 13:55:33.690915 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.690048    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/08545f35-761a-4929-9192-254ca6b76569-node-log\") pod \"ovnkube-node-6546h\" (UID: \"08545f35-761a-4929-9192-254ca6b76569\") " pod="openshift-ovn-kubernetes/ovnkube-node-6546h"
Sep 17 13:55:33.690915 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.690057    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08545f35-761a-4929-9192-254ca6b76569-etc-openvswitch\") pod \"ovnkube-node-6546h\" (UID: \"08545f35-761a-4929-9192-254ca6b76569\") " pod="openshift-ovn-kubernetes/ovnkube-node-6546h"
Sep 17 13:55:33.690915 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.690091    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/08545f35-761a-4929-9192-254ca6b76569-run-ovn\") pod \"ovnkube-node-6546h\" (UID: \"08545f35-761a-4929-9192-254ca6b76569\") " pod="openshift-ovn-kubernetes/ovnkube-node-6546h"
Sep 17 13:55:33.690915 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.690308    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ece625b2-eada-4304-a7e0-c96a322f289b-cni-binary-copy\") pod \"multus-vqb5r\" (UID: \"ece625b2-eada-4304-a7e0-c96a322f289b\") " pod="openshift-multus/multus-vqb5r"
Sep 17 13:55:33.690915 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.690310    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ece625b2-eada-4304-a7e0-c96a322f289b-multus-daemon-config\") pod \"multus-vqb5r\" (UID: \"ece625b2-eada-4304-a7e0-c96a322f289b\") " pod="openshift-multus/multus-vqb5r"
Sep 17 13:55:33.690915 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.690349    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-575ht\" (UniqueName: \"kubernetes.io/projected/ece625b2-eada-4304-a7e0-c96a322f289b-kube-api-access-575ht\") pod \"multus-vqb5r\" (UID: \"ece625b2-eada-4304-a7e0-c96a322f289b\") " pod="openshift-multus/multus-vqb5r"
Sep 17 13:55:33.690915 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.690368    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ece625b2-eada-4304-a7e0-c96a322f289b-etc-kubernetes\") pod \"multus-vqb5r\" (UID: \"ece625b2-eada-4304-a7e0-c96a322f289b\") " pod="openshift-multus/multus-vqb5r"
Sep 17 13:55:33.690915 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.690379    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/08545f35-761a-4929-9192-254ca6b76569-run-systemd\") pod \"ovnkube-node-6546h\" (UID: \"08545f35-761a-4929-9192-254ca6b76569\") " pod="openshift-ovn-kubernetes/ovnkube-node-6546h"
Sep 17 13:55:33.691901 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.690414    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/08545f35-761a-4929-9192-254ca6b76569-run-systemd\") pod \"ovnkube-node-6546h\" (UID: \"08545f35-761a-4929-9192-254ca6b76569\") " pod="openshift-ovn-kubernetes/ovnkube-node-6546h"
Sep 17 13:55:33.691901 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.690420    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ece625b2-eada-4304-a7e0-c96a322f289b-host-run-k8s-cni-cncf-io\") pod \"multus-vqb5r\" (UID: \"ece625b2-eada-4304-a7e0-c96a322f289b\") " pod="openshift-multus/multus-vqb5r"
Sep 17 13:55:33.691901 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.690419    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a4a8010a-d12b-4b1a-a0db-e1c64109b151-hosts-file\") pod \"node-resolver-2zqvg\" (UID: \"a4a8010a-d12b-4b1a-a0db-e1c64109b151\") " pod="openshift-dns/node-resolver-2zqvg"
Sep 17 13:55:33.691901 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.690453    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ece625b2-eada-4304-a7e0-c96a322f289b-hostroot\") pod \"multus-vqb5r\" (UID: \"ece625b2-eada-4304-a7e0-c96a322f289b\") " pod="openshift-multus/multus-vqb5r"
Sep 17 13:55:33.691901 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.690480    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/14057e5c-c9fb-4894-869c-9780abc9c769-original-pull-secret\") pod \"global-pull-secret-syncer-58g8x\" (UID: \"14057e5c-c9fb-4894-869c-9780abc9c769\") " pod="kube-system/global-pull-secret-syncer-58g8x"
Sep 17 13:55:33.691901 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.690487    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ece625b2-eada-4304-a7e0-c96a322f289b-host-run-k8s-cni-cncf-io\") pod \"multus-vqb5r\" (UID: \"ece625b2-eada-4304-a7e0-c96a322f289b\") " pod="openshift-multus/multus-vqb5r"
Sep 17 13:55:33.691901 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.690512    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4071dbb7-7923-4beb-856a-f03b5c7cfb98-host-slash\") pod \"iptables-alerter-xrrk6\" (UID: \"4071dbb7-7923-4beb-856a-f03b5c7cfb98\") " pod="openshift-network-operator/iptables-alerter-xrrk6"
Sep 17 13:55:33.691901 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.690487    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/08545f35-761a-4929-9192-254ca6b76569-log-socket\") pod \"ovnkube-node-6546h\" (UID: \"08545f35-761a-4929-9192-254ca6b76569\") " pod="openshift-ovn-kubernetes/ovnkube-node-6546h"
Sep 17 13:55:33.691901 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.690512    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/08545f35-761a-4929-9192-254ca6b76569-env-overrides\") pod \"ovnkube-node-6546h\" (UID: \"08545f35-761a-4929-9192-254ca6b76569\") " pod="openshift-ovn-kubernetes/ovnkube-node-6546h"
Sep 17 13:55:33.691901 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.690574    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/14057e5c-c9fb-4894-869c-9780abc9c769-kubelet-config\") pod \"global-pull-secret-syncer-58g8x\" (UID: \"14057e5c-c9fb-4894-869c-9780abc9c769\") " pod="kube-system/global-pull-secret-syncer-58g8x"
Sep 17 13:55:33.691901 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.690583    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ece625b2-eada-4304-a7e0-c96a322f289b-host-run-multus-certs\") pod \"multus-vqb5r\" (UID: \"ece625b2-eada-4304-a7e0-c96a322f289b\") " pod="openshift-multus/multus-vqb5r"
Sep 17 13:55:33.691901 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.690574    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d12b5b83-268b-404c-b77b-dd8ec7ea24d1-metrics-certs\") pod \"network-metrics-daemon-dqcvb\" (UID: \"d12b5b83-268b-404c-b77b-dd8ec7ea24d1\") " pod="openshift-multus/network-metrics-daemon-dqcvb"
Sep 17 13:55:33.691901 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:33.690596    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 13:55:33.691901 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.690621    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4071dbb7-7923-4beb-856a-f03b5c7cfb98-host-slash\") pod \"iptables-alerter-xrrk6\" (UID: \"4071dbb7-7923-4beb-856a-f03b5c7cfb98\") " pod="openshift-network-operator/iptables-alerter-xrrk6"
Sep 17 13:55:33.691901 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.690516    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ece625b2-eada-4304-a7e0-c96a322f289b-host-var-lib-cni-multus\") pod \"multus-vqb5r\" (UID: \"ece625b2-eada-4304-a7e0-c96a322f289b\") " pod="openshift-multus/multus-vqb5r"
Sep 17 13:55:33.691901 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.690659    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ece625b2-eada-4304-a7e0-c96a322f289b-host-var-lib-cni-bin\") pod \"multus-vqb5r\" (UID: \"ece625b2-eada-4304-a7e0-c96a322f289b\") " pod="openshift-multus/multus-vqb5r"
Sep 17 13:55:33.691901 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.690664    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/08545f35-761a-4929-9192-254ca6b76569-host-slash\") pod \"ovnkube-node-6546h\" (UID: \"08545f35-761a-4929-9192-254ca6b76569\") " pod="openshift-ovn-kubernetes/ovnkube-node-6546h"
Sep 17 13:55:33.691901 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.690623    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/08545f35-761a-4929-9192-254ca6b76569-host-slash\") pod \"ovnkube-node-6546h\" (UID: \"08545f35-761a-4929-9192-254ca6b76569\") " pod="openshift-ovn-kubernetes/ovnkube-node-6546h"
Sep 17 13:55:33.692744 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:33.690670    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14057e5c-c9fb-4894-869c-9780abc9c769-original-pull-secret podName:14057e5c-c9fb-4894-869c-9780abc9c769 nodeName:}" failed. No retries permitted until 2026-09-17 13:55:34.190651356 +0000 UTC m=+3.132794997 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/14057e5c-c9fb-4894-869c-9780abc9c769-original-pull-secret") pod "global-pull-secret-syncer-58g8x" (UID: "14057e5c-c9fb-4894-869c-9780abc9c769") : object "kube-system"/"original-pull-secret" not registered
Sep 17 13:55:33.692744 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.690715    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/08545f35-761a-4929-9192-254ca6b76569-ovnkube-script-lib\") pod \"ovnkube-node-6546h\" (UID: \"08545f35-761a-4929-9192-254ca6b76569\") " pod="openshift-ovn-kubernetes/ovnkube-node-6546h"
Sep 17 13:55:33.692744 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:33.690730    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 13:55:33.692744 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.690741    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ece625b2-eada-4304-a7e0-c96a322f289b-system-cni-dir\") pod \"multus-vqb5r\" (UID: \"ece625b2-eada-4304-a7e0-c96a322f289b\") " pod="openshift-multus/multus-vqb5r"
Sep 17 13:55:33.692744 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.690758    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/eae24b6c-3322-4165-96fa-cd61dab923ca-konnectivity-ca\") pod \"konnectivity-agent-gqsqk\" (UID: \"eae24b6c-3322-4165-96fa-cd61dab923ca\") " pod="kube-system/konnectivity-agent-gqsqk"
Sep 17 13:55:33.692744 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.690771    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ece625b2-eada-4304-a7e0-c96a322f289b-multus-cni-dir\") pod \"multus-vqb5r\" (UID: \"ece625b2-eada-4304-a7e0-c96a322f289b\") " pod="openshift-multus/multus-vqb5r"
Sep 17 13:55:33.692744 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:33.690780    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d12b5b83-268b-404c-b77b-dd8ec7ea24d1-metrics-certs podName:d12b5b83-268b-404c-b77b-dd8ec7ea24d1 nodeName:}" failed. No retries permitted until 2026-09-17 13:55:34.190766331 +0000 UTC m=+3.132909957 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d12b5b83-268b-404c-b77b-dd8ec7ea24d1-metrics-certs") pod "network-metrics-daemon-dqcvb" (UID: "d12b5b83-268b-404c-b77b-dd8ec7ea24d1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 13:55:33.692744 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.690805    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ece625b2-eada-4304-a7e0-c96a322f289b-hostroot\") pod \"multus-vqb5r\" (UID: \"ece625b2-eada-4304-a7e0-c96a322f289b\") " pod="openshift-multus/multus-vqb5r"
Sep 17 13:55:33.692744 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.690825    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ece625b2-eada-4304-a7e0-c96a322f289b-multus-cni-dir\") pod \"multus-vqb5r\" (UID: \"ece625b2-eada-4304-a7e0-c96a322f289b\") " pod="openshift-multus/multus-vqb5r"
Sep 17 13:55:33.692744 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.690816    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ece625b2-eada-4304-a7e0-c96a322f289b-multus-socket-dir-parent\") pod \"multus-vqb5r\" (UID: \"ece625b2-eada-4304-a7e0-c96a322f289b\") " pod="openshift-multus/multus-vqb5r"
Sep 17 13:55:33.692744 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.690878    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9fcb9893-714e-47b5-9b6f-37e706357ffe-serviceca\") pod \"node-ca-9wpxw\" (UID: \"9fcb9893-714e-47b5-9b6f-37e706357ffe\") " pod="openshift-image-registry/node-ca-9wpxw"
Sep 17 13:55:33.692744 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.690899    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08545f35-761a-4929-9192-254ca6b76569-run-openvswitch\") pod \"ovnkube-node-6546h\" (UID: \"08545f35-761a-4929-9192-254ca6b76569\") " pod="openshift-ovn-kubernetes/ovnkube-node-6546h"
Sep 17 13:55:33.692744 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.690933    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ece625b2-eada-4304-a7e0-c96a322f289b-multus-socket-dir-parent\") pod \"multus-vqb5r\" (UID: \"ece625b2-eada-4304-a7e0-c96a322f289b\") " pod="openshift-multus/multus-vqb5r"
Sep 17 13:55:33.692744 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.690943    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/08545f35-761a-4929-9192-254ca6b76569-ovnkube-config\") pod \"ovnkube-node-6546h\" (UID: \"08545f35-761a-4929-9192-254ca6b76569\") " pod="openshift-ovn-kubernetes/ovnkube-node-6546h"
Sep 17 13:55:33.692744 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.690879    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ece625b2-eada-4304-a7e0-c96a322f289b-system-cni-dir\") pod \"multus-vqb5r\" (UID: \"ece625b2-eada-4304-a7e0-c96a322f289b\") " pod="openshift-multus/multus-vqb5r"
Sep 17 13:55:33.692744 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.690977    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9fcb9893-714e-47b5-9b6f-37e706357ffe-host\") pod \"node-ca-9wpxw\" (UID: \"9fcb9893-714e-47b5-9b6f-37e706357ffe\") " pod="openshift-image-registry/node-ca-9wpxw"
Sep 17 13:55:33.692744 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.690985    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08545f35-761a-4929-9192-254ca6b76569-run-openvswitch\") pod \"ovnkube-node-6546h\" (UID: \"08545f35-761a-4929-9192-254ca6b76569\") " pod="openshift-ovn-kubernetes/ovnkube-node-6546h"
Sep 17 13:55:33.693405 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.691004    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2tbt\" (UniqueName: \"kubernetes.io/projected/9fcb9893-714e-47b5-9b6f-37e706357ffe-kube-api-access-x2tbt\") pod \"node-ca-9wpxw\" (UID: \"9fcb9893-714e-47b5-9b6f-37e706357ffe\") " pod="openshift-image-registry/node-ca-9wpxw"
Sep 17 13:55:33.693405 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.691028    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9fcb9893-714e-47b5-9b6f-37e706357ffe-host\") pod \"node-ca-9wpxw\" (UID: \"9fcb9893-714e-47b5-9b6f-37e706357ffe\") " pod="openshift-image-registry/node-ca-9wpxw"
Sep 17 13:55:33.693405 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.691030    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8xxr\" (UniqueName: \"kubernetes.io/projected/a4a8010a-d12b-4b1a-a0db-e1c64109b151-kube-api-access-f8xxr\") pod \"node-resolver-2zqvg\" (UID: \"a4a8010a-d12b-4b1a-a0db-e1c64109b151\") " pod="openshift-dns/node-resolver-2zqvg"
Sep 17 13:55:33.693405 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.691064    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/eae24b6c-3322-4165-96fa-cd61dab923ca-agent-certs\") pod \"konnectivity-agent-gqsqk\" (UID: \"eae24b6c-3322-4165-96fa-cd61dab923ca\") " pod="kube-system/konnectivity-agent-gqsqk"
Sep 17 13:55:33.693405 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.691091    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4a8010a-d12b-4b1a-a0db-e1c64109b151-tmp-dir\") pod \"node-resolver-2zqvg\" (UID: \"a4a8010a-d12b-4b1a-a0db-e1c64109b151\") " pod="openshift-dns/node-resolver-2zqvg"
Sep 17 13:55:33.693405 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.691118    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4071dbb7-7923-4beb-856a-f03b5c7cfb98-iptables-alerter-script\") pod \"iptables-alerter-xrrk6\" (UID: \"4071dbb7-7923-4beb-856a-f03b5c7cfb98\") " pod="openshift-network-operator/iptables-alerter-xrrk6"
Sep 17 13:55:33.693405 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.691158    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/08545f35-761a-4929-9192-254ca6b76569-systemd-units\") pod \"ovnkube-node-6546h\" (UID: \"08545f35-761a-4929-9192-254ca6b76569\") " pod="openshift-ovn-kubernetes/ovnkube-node-6546h"
Sep 17 13:55:33.693405 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.691177    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/08545f35-761a-4929-9192-254ca6b76569-ovnkube-script-lib\") pod \"ovnkube-node-6546h\" (UID: \"08545f35-761a-4929-9192-254ca6b76569\") " pod="openshift-ovn-kubernetes/ovnkube-node-6546h"
Sep 17 13:55:33.693405 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.691231    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/08545f35-761a-4929-9192-254ca6b76569-systemd-units\") pod \"ovnkube-node-6546h\" (UID: \"08545f35-761a-4929-9192-254ca6b76569\") " pod="openshift-ovn-kubernetes/ovnkube-node-6546h"
Sep 17 13:55:33.693405 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.691394    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9fcb9893-714e-47b5-9b6f-37e706357ffe-serviceca\") pod \"node-ca-9wpxw\" (UID: \"9fcb9893-714e-47b5-9b6f-37e706357ffe\") " pod="openshift-image-registry/node-ca-9wpxw"
Sep 17 13:55:33.693405 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.691473    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/08545f35-761a-4929-9192-254ca6b76569-ovnkube-config\") pod \"ovnkube-node-6546h\" (UID: \"08545f35-761a-4929-9192-254ca6b76569\") " pod="openshift-ovn-kubernetes/ovnkube-node-6546h"
Sep 17 13:55:33.693405 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.691607    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4a8010a-d12b-4b1a-a0db-e1c64109b151-tmp-dir\") pod \"node-resolver-2zqvg\" (UID: \"a4a8010a-d12b-4b1a-a0db-e1c64109b151\") " pod="openshift-dns/node-resolver-2zqvg"
Sep 17 13:55:33.693405 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.691816    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4071dbb7-7923-4beb-856a-f03b5c7cfb98-iptables-alerter-script\") pod \"iptables-alerter-xrrk6\" (UID: \"4071dbb7-7923-4beb-856a-f03b5c7cfb98\") " pod="openshift-network-operator/iptables-alerter-xrrk6"
Sep 17 13:55:33.694336 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.694302    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/eae24b6c-3322-4165-96fa-cd61dab923ca-agent-certs\") pod \"konnectivity-agent-gqsqk\" (UID: \"eae24b6c-3322-4165-96fa-cd61dab923ca\") " pod="kube-system/konnectivity-agent-gqsqk"
Sep 17 13:55:33.694551 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.694519    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/08545f35-761a-4929-9192-254ca6b76569-ovn-node-metrics-cert\") pod \"ovnkube-node-6546h\" (UID: \"08545f35-761a-4929-9192-254ca6b76569\") " pod="openshift-ovn-kubernetes/ovnkube-node-6546h"
Sep 17 13:55:33.711450 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.711423    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gtf9\" (UniqueName: \"kubernetes.io/projected/08545f35-761a-4929-9192-254ca6b76569-kube-api-access-4gtf9\") pod \"ovnkube-node-6546h\" (UID: \"08545f35-761a-4929-9192-254ca6b76569\") " pod="openshift-ovn-kubernetes/ovnkube-node-6546h"
Sep 17 13:55:33.714056 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.714031    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2tbt\" (UniqueName: \"kubernetes.io/projected/9fcb9893-714e-47b5-9b6f-37e706357ffe-kube-api-access-x2tbt\") pod \"node-ca-9wpxw\" (UID: \"9fcb9893-714e-47b5-9b6f-37e706357ffe\") " pod="openshift-image-registry/node-ca-9wpxw"
Sep 17 13:55:33.714880 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.714858    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-575ht\" (UniqueName: \"kubernetes.io/projected/ece625b2-eada-4304-a7e0-c96a322f289b-kube-api-access-575ht\") pod \"multus-vqb5r\" (UID: \"ece625b2-eada-4304-a7e0-c96a322f289b\") " pod="openshift-multus/multus-vqb5r"
Sep 17 13:55:33.715455 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.715412    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8xxr\" (UniqueName: \"kubernetes.io/projected/a4a8010a-d12b-4b1a-a0db-e1c64109b151-kube-api-access-f8xxr\") pod \"node-resolver-2zqvg\" (UID: \"a4a8010a-d12b-4b1a-a0db-e1c64109b151\") " pod="openshift-dns/node-resolver-2zqvg"
Sep 17 13:55:33.716083 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.716041    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nw7k\" (UniqueName: \"kubernetes.io/projected/d12b5b83-268b-404c-b77b-dd8ec7ea24d1-kube-api-access-9nw7k\") pod \"network-metrics-daemon-dqcvb\" (UID: \"d12b5b83-268b-404c-b77b-dd8ec7ea24d1\") " pod="openshift-multus/network-metrics-daemon-dqcvb"
Sep 17 13:55:33.716912 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.716870    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l695d\" (UniqueName: \"kubernetes.io/projected/4071dbb7-7923-4beb-856a-f03b5c7cfb98-kube-api-access-l695d\") pod \"iptables-alerter-xrrk6\" (UID: \"4071dbb7-7923-4beb-856a-f03b5c7cfb98\") " pod="openshift-network-operator/iptables-alerter-xrrk6"
Sep 17 13:55:33.779808 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.779781    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ssnxx"
Sep 17 13:55:33.791234 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.791041    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6gqm"
Sep 17 13:55:33.803151 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.803128    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zxhg5"
Sep 17 13:55:33.809278 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.809259    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2zqvg"
Sep 17 13:55:33.818088 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.818063    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xrrk6"
Sep 17 13:55:33.826008 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.825992    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6546h"
Sep 17 13:55:33.831982 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.831962    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vqb5r"
Sep 17 13:55:33.838749 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.838730    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gqsqk"
Sep 17 13:55:33.842825 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:33.842806    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9wpxw"
Sep 17 13:55:34.194816 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:34.194784    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/14057e5c-c9fb-4894-869c-9780abc9c769-original-pull-secret\") pod \"global-pull-secret-syncer-58g8x\" (UID: \"14057e5c-c9fb-4894-869c-9780abc9c769\") " pod="kube-system/global-pull-secret-syncer-58g8x"
Sep 17 13:55:34.195017 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:34.194832    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d12b5b83-268b-404c-b77b-dd8ec7ea24d1-metrics-certs\") pod \"network-metrics-daemon-dqcvb\" (UID: \"d12b5b83-268b-404c-b77b-dd8ec7ea24d1\") " pod="openshift-multus/network-metrics-daemon-dqcvb"
Sep 17 13:55:34.195017 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:34.194862    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rndqv\" (UniqueName: \"kubernetes.io/projected/8677b05f-e355-4c69-b16e-8e64d8b0e701-kube-api-access-rndqv\") pod \"network-check-target-9jwwx\" (UID: \"8677b05f-e355-4c69-b16e-8e64d8b0e701\") " pod="openshift-network-diagnostics/network-check-target-9jwwx"
Sep 17 13:55:34.195017 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:34.194962    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 13:55:34.195162 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:34.195039    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14057e5c-c9fb-4894-869c-9780abc9c769-original-pull-secret podName:14057e5c-c9fb-4894-869c-9780abc9c769 nodeName:}" failed. No retries permitted until 2026-09-17 13:55:35.195016805 +0000 UTC m=+4.137160454 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/14057e5c-c9fb-4894-869c-9780abc9c769-original-pull-secret") pod "global-pull-secret-syncer-58g8x" (UID: "14057e5c-c9fb-4894-869c-9780abc9c769") : object "kube-system"/"original-pull-secret" not registered
Sep 17 13:55:34.195162 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:34.194965    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 13:55:34.195162 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:34.194967    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 13:55:34.195162 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:34.195141    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 13:55:34.195162 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:34.195157    2560 projected.go:196] Error preparing data for projected volume kube-api-access-rndqv for pod openshift-network-diagnostics/network-check-target-9jwwx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 13:55:34.195162 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:34.195115    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d12b5b83-268b-404c-b77b-dd8ec7ea24d1-metrics-certs podName:d12b5b83-268b-404c-b77b-dd8ec7ea24d1 nodeName:}" failed. No retries permitted until 2026-09-17 13:55:35.195102051 +0000 UTC m=+4.137245691 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d12b5b83-268b-404c-b77b-dd8ec7ea24d1-metrics-certs") pod "network-metrics-daemon-dqcvb" (UID: "d12b5b83-268b-404c-b77b-dd8ec7ea24d1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 13:55:34.195412 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:34.195209    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8677b05f-e355-4c69-b16e-8e64d8b0e701-kube-api-access-rndqv podName:8677b05f-e355-4c69-b16e-8e64d8b0e701 nodeName:}" failed. No retries permitted until 2026-09-17 13:55:35.195193956 +0000 UTC m=+4.137337602 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-rndqv" (UniqueName: "kubernetes.io/projected/8677b05f-e355-4c69-b16e-8e64d8b0e701-kube-api-access-rndqv") pod "network-check-target-9jwwx" (UID: "8677b05f-e355-4c69-b16e-8e64d8b0e701") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 13:55:34.536587 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:34.536490    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 13:50:34 +0000 UTC" deadline="2028-04-01 15:59:58.722568575 +0000 UTC"
Sep 17 13:55:34.536587 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:34.536528    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13490h4m24.186045453s"
Sep 17 13:55:34.778090 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:34.778054    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4a8010a_d12b_4b1a_a0db_e1c64109b151.slice/crio-5e14c1804c5678a0caf3b33519145685275f3adb8729575e5bc061c2550cc670 WatchSource:0}: Error finding container 5e14c1804c5678a0caf3b33519145685275f3adb8729575e5bc061c2550cc670: Status 404 returned error can't find the container with id 5e14c1804c5678a0caf3b33519145685275f3adb8729575e5bc061c2550cc670
Sep 17 13:55:34.801091 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:34.801054    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeae24b6c_3322_4165_96fa_cd61dab923ca.slice/crio-c50b8d913b3475ade25464cc67648c65bd81e04d8f082030c929419fb0ee19c1 WatchSource:0}: Error finding container c50b8d913b3475ade25464cc67648c65bd81e04d8f082030c929419fb0ee19c1: Status 404 returned error can't find the container with id c50b8d913b3475ade25464cc67648c65bd81e04d8f082030c929419fb0ee19c1
Sep 17 13:55:34.802327 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:34.802076    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod714b66d5_d63d_4ac3_a097_f338deef9592.slice/crio-0638eb37370ef1d06038317a4845e9a84b64b9145304cccafab93057ebacfcf4 WatchSource:0}: Error finding container 0638eb37370ef1d06038317a4845e9a84b64b9145304cccafab93057ebacfcf4: Status 404 returned error can't find the container with id 0638eb37370ef1d06038317a4845e9a84b64b9145304cccafab93057ebacfcf4
Sep 17 13:55:34.802849 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:34.802820    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fcb9893_714e_47b5_9b6f_37e706357ffe.slice/crio-c70280c6dee9f03bdf6d6caa7db6cbf5115b1453e041aac7262763cb3f059e70 WatchSource:0}: Error finding container c70280c6dee9f03bdf6d6caa7db6cbf5115b1453e041aac7262763cb3f059e70: Status 404 returned error can't find the container with id c70280c6dee9f03bdf6d6caa7db6cbf5115b1453e041aac7262763cb3f059e70
Sep 17 13:55:34.807183 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:34.804201    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2ae1c9d_26b0_47e9_ae74_53904cc9f089.slice/crio-4a64df5ec82d545b7a69b7d84b067e0ef35873052f4069e4a51a92878db4d5fb WatchSource:0}: Error finding container 4a64df5ec82d545b7a69b7d84b067e0ef35873052f4069e4a51a92878db4d5fb: Status 404 returned error can't find the container with id 4a64df5ec82d545b7a69b7d84b067e0ef35873052f4069e4a51a92878db4d5fb
Sep 17 13:55:34.811890 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:34.811867    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4071dbb7_7923_4beb_856a_f03b5c7cfb98.slice/crio-9bbbaa81ae2f898cf44025b0a154403336bdbe84616a895847835b85be2b6a1f WatchSource:0}: Error finding container 9bbbaa81ae2f898cf44025b0a154403336bdbe84616a895847835b85be2b6a1f: Status 404 returned error can't find the container with id 9bbbaa81ae2f898cf44025b0a154403336bdbe84616a895847835b85be2b6a1f
Sep 17 13:55:34.813066 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:34.813041    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08545f35_761a_4929_9192_254ca6b76569.slice/crio-b7a1ebd87bdcfbb8a8974beac6a5ddb284ef564febadbd529d55327fbc6f5d20 WatchSource:0}: Error finding container b7a1ebd87bdcfbb8a8974beac6a5ddb284ef564febadbd529d55327fbc6f5d20: Status 404 returned error can't find the container with id b7a1ebd87bdcfbb8a8974beac6a5ddb284ef564febadbd529d55327fbc6f5d20
Sep 17 13:55:34.815693 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:55:34.814052    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podece625b2_eada_4304_a7e0_c96a322f289b.slice/crio-50291059f035ffd1b7277c4b5c793777d8b0da6d1084b8ff8f510414847bb8d5 WatchSource:0}: Error finding container 50291059f035ffd1b7277c4b5c793777d8b0da6d1084b8ff8f510414847bb8d5: Status 404 returned error can't find the container with id 50291059f035ffd1b7277c4b5c793777d8b0da6d1084b8ff8f510414847bb8d5
Sep 17 13:55:35.202800 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:35.202759    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rndqv\" (UniqueName: \"kubernetes.io/projected/8677b05f-e355-4c69-b16e-8e64d8b0e701-kube-api-access-rndqv\") pod \"network-check-target-9jwwx\" (UID: \"8677b05f-e355-4c69-b16e-8e64d8b0e701\") " pod="openshift-network-diagnostics/network-check-target-9jwwx"
Sep 17 13:55:35.202937 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:35.202916    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/14057e5c-c9fb-4894-869c-9780abc9c769-original-pull-secret\") pod \"global-pull-secret-syncer-58g8x\" (UID: \"14057e5c-c9fb-4894-869c-9780abc9c769\") " pod="kube-system/global-pull-secret-syncer-58g8x"
Sep 17 13:55:35.202997 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:35.202953    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d12b5b83-268b-404c-b77b-dd8ec7ea24d1-metrics-certs\") pod \"network-metrics-daemon-dqcvb\" (UID: \"d12b5b83-268b-404c-b77b-dd8ec7ea24d1\") " pod="openshift-multus/network-metrics-daemon-dqcvb"
Sep 17 13:55:35.202997 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:35.202925    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 13:55:35.203106 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:35.202998    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 13:55:35.203106 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:35.203026    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 13:55:35.203106 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:35.203034    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 13:55:35.203106 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:35.203043    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14057e5c-c9fb-4894-869c-9780abc9c769-original-pull-secret podName:14057e5c-c9fb-4894-869c-9780abc9c769 nodeName:}" failed. No retries permitted until 2026-09-17 13:55:37.203030872 +0000 UTC m=+6.145174498 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/14057e5c-c9fb-4894-869c-9780abc9c769-original-pull-secret") pod "global-pull-secret-syncer-58g8x" (UID: "14057e5c-c9fb-4894-869c-9780abc9c769") : object "kube-system"/"original-pull-secret" not registered
Sep 17 13:55:35.203106 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:35.203047    2560 projected.go:196] Error preparing data for projected volume kube-api-access-rndqv for pod openshift-network-diagnostics/network-check-target-9jwwx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 13:55:35.203106 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:35.203087    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d12b5b83-268b-404c-b77b-dd8ec7ea24d1-metrics-certs podName:d12b5b83-268b-404c-b77b-dd8ec7ea24d1 nodeName:}" failed. No retries permitted until 2026-09-17 13:55:37.203063878 +0000 UTC m=+6.145207504 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d12b5b83-268b-404c-b77b-dd8ec7ea24d1-metrics-certs") pod "network-metrics-daemon-dqcvb" (UID: "d12b5b83-268b-404c-b77b-dd8ec7ea24d1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 13:55:35.203106 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:35.203106    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8677b05f-e355-4c69-b16e-8e64d8b0e701-kube-api-access-rndqv podName:8677b05f-e355-4c69-b16e-8e64d8b0e701 nodeName:}" failed. No retries permitted until 2026-09-17 13:55:37.203095167 +0000 UTC m=+6.145238792 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-rndqv" (UniqueName: "kubernetes.io/projected/8677b05f-e355-4c69-b16e-8e64d8b0e701-kube-api-access-rndqv") pod "network-check-target-9jwwx" (UID: "8677b05f-e355-4c69-b16e-8e64d8b0e701") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 13:55:35.537256 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:35.537135    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 13:50:34 +0000 UTC" deadline="2028-03-18 20:33:21.850579203 +0000 UTC"
Sep 17 13:55:35.537256 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:35.537221    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13158h37m46.313374757s"
Sep 17 13:55:35.566707 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:35.566675    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9jwwx"
Sep 17 13:55:35.566878 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:35.566815    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9jwwx" podUID="8677b05f-e355-4c69-b16e-8e64d8b0e701"
Sep 17 13:55:35.567258 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:35.567234    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqcvb"
Sep 17 13:55:35.567379 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:35.567359    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqcvb" podUID="d12b5b83-268b-404c-b77b-dd8ec7ea24d1"
Sep 17 13:55:35.567449 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:35.567437    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58g8x"
Sep 17 13:55:35.567560 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:35.567514    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-58g8x" podUID="14057e5c-c9fb-4894-869c-9780abc9c769"
Sep 17 13:55:35.576153 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:35.576119    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zxhg5" event={"ID":"bfe7d6ff-377c-4449-8c3c-69163deda18c","Type":"ContainerStarted","Data":"b4ebf621832d3cc73564a579dd4e67e2f4162e37767ec0a77df7a1d3c22951e1"}
Sep 17 13:55:35.578403 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:35.578372    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vqb5r" event={"ID":"ece625b2-eada-4304-a7e0-c96a322f289b","Type":"ContainerStarted","Data":"50291059f035ffd1b7277c4b5c793777d8b0da6d1084b8ff8f510414847bb8d5"}
Sep 17 13:55:35.581984 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:35.581953    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6gqm" event={"ID":"714b66d5-d63d-4ac3-a097-f338deef9592","Type":"ContainerStarted","Data":"0638eb37370ef1d06038317a4845e9a84b64b9145304cccafab93057ebacfcf4"}
Sep 17 13:55:35.586384 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:35.586334    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gqsqk" event={"ID":"eae24b6c-3322-4165-96fa-cd61dab923ca","Type":"ContainerStarted","Data":"c50b8d913b3475ade25464cc67648c65bd81e04d8f082030c929419fb0ee19c1"}
Sep 17 13:55:35.603142 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:35.603112    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-158.ec2.internal" event={"ID":"9a84bb5c5cbac6dbd226621c20c98385","Type":"ContainerStarted","Data":"308dd299bc327f33459e1b6956bfabf9731d54aaa24d264886c36d92b8cd6c14"}
Sep 17 13:55:35.611647 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:35.611617    2560 generic.go:356] "Generic (PLEG): container finished" podID="201ffe21c4e43373b0c674db075b035e" containerID="9a83c69002ea48a18c5134a3ee479f711863b65222a69841cb6c11906ddb7cdc" exitCode=0
Sep 17 13:55:35.612172 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:35.612149    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-158.ec2.internal" event={"ID":"201ffe21c4e43373b0c674db075b035e","Type":"ContainerDied","Data":"9a83c69002ea48a18c5134a3ee479f711863b65222a69841cb6c11906ddb7cdc"}
Sep 17 13:55:35.628002 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:35.627958    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6546h" event={"ID":"08545f35-761a-4929-9192-254ca6b76569","Type":"ContainerStarted","Data":"b7a1ebd87bdcfbb8a8974beac6a5ddb284ef564febadbd529d55327fbc6f5d20"}
Sep 17 13:55:35.636152 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:35.636100    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-158.ec2.internal" podStartSLOduration=2.63608217 podStartE2EDuration="2.63608217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 13:55:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 13:55:35.620732924 +0000 UTC m=+4.562876573" watchObservedRunningTime="2026-09-17 13:55:35.63608217 +0000 UTC m=+4.578225821"
Sep 17 13:55:35.638620 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:35.638593    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xrrk6" event={"ID":"4071dbb7-7923-4beb-856a-f03b5c7cfb98","Type":"ContainerStarted","Data":"9bbbaa81ae2f898cf44025b0a154403336bdbe84616a895847835b85be2b6a1f"}
Sep 17 13:55:35.653633 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:35.653092    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ssnxx" event={"ID":"e2ae1c9d-26b0-47e9-ae74-53904cc9f089","Type":"ContainerStarted","Data":"4a64df5ec82d545b7a69b7d84b067e0ef35873052f4069e4a51a92878db4d5fb"}
Sep 17 13:55:35.664264 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:35.664237    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9wpxw" event={"ID":"9fcb9893-714e-47b5-9b6f-37e706357ffe","Type":"ContainerStarted","Data":"c70280c6dee9f03bdf6d6caa7db6cbf5115b1453e041aac7262763cb3f059e70"}
Sep 17 13:55:35.678915 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:35.678889    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2zqvg" event={"ID":"a4a8010a-d12b-4b1a-a0db-e1c64109b151","Type":"ContainerStarted","Data":"5e14c1804c5678a0caf3b33519145685275f3adb8729575e5bc061c2550cc670"}
Sep 17 13:55:36.696214 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:36.696176    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-158.ec2.internal" event={"ID":"201ffe21c4e43373b0c674db075b035e","Type":"ContainerStarted","Data":"daa6659d57db71ecb257a7659819ecb88c2bee53af55dc7452c4b2f9b634a82f"}
Sep 17 13:55:36.712348 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:36.712292    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-158.ec2.internal" podStartSLOduration=3.712271195 podStartE2EDuration="3.712271195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 13:55:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 13:55:36.711466878 +0000 UTC m=+5.653610530" watchObservedRunningTime="2026-09-17 13:55:36.712271195 +0000 UTC m=+5.654414845"
Sep 17 13:55:37.220586 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:37.220547    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rndqv\" (UniqueName: \"kubernetes.io/projected/8677b05f-e355-4c69-b16e-8e64d8b0e701-kube-api-access-rndqv\") pod \"network-check-target-9jwwx\" (UID: \"8677b05f-e355-4c69-b16e-8e64d8b0e701\") " pod="openshift-network-diagnostics/network-check-target-9jwwx"
Sep 17 13:55:37.220769 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:37.220646    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/14057e5c-c9fb-4894-869c-9780abc9c769-original-pull-secret\") pod \"global-pull-secret-syncer-58g8x\" (UID: \"14057e5c-c9fb-4894-869c-9780abc9c769\") " pod="kube-system/global-pull-secret-syncer-58g8x"
Sep 17 13:55:37.220769 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:37.220679    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d12b5b83-268b-404c-b77b-dd8ec7ea24d1-metrics-certs\") pod \"network-metrics-daemon-dqcvb\" (UID: \"d12b5b83-268b-404c-b77b-dd8ec7ea24d1\") " pod="openshift-multus/network-metrics-daemon-dqcvb"
Sep 17 13:55:37.220892 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:37.220836    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 13:55:37.220945 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:37.220905    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d12b5b83-268b-404c-b77b-dd8ec7ea24d1-metrics-certs podName:d12b5b83-268b-404c-b77b-dd8ec7ea24d1 nodeName:}" failed. No retries permitted until 2026-09-17 13:55:41.220882861 +0000 UTC m=+10.163026493 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d12b5b83-268b-404c-b77b-dd8ec7ea24d1-metrics-certs") pod "network-metrics-daemon-dqcvb" (UID: "d12b5b83-268b-404c-b77b-dd8ec7ea24d1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 13:55:37.221397 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:37.221375    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 13:55:37.221483 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:37.221404    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 13:55:37.221483 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:37.221419    2560 projected.go:196] Error preparing data for projected volume kube-api-access-rndqv for pod openshift-network-diagnostics/network-check-target-9jwwx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 13:55:37.221483 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:37.221466    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8677b05f-e355-4c69-b16e-8e64d8b0e701-kube-api-access-rndqv podName:8677b05f-e355-4c69-b16e-8e64d8b0e701 nodeName:}" failed. No retries permitted until 2026-09-17 13:55:41.221451699 +0000 UTC m=+10.163595333 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-rndqv" (UniqueName: "kubernetes.io/projected/8677b05f-e355-4c69-b16e-8e64d8b0e701-kube-api-access-rndqv") pod "network-check-target-9jwwx" (UID: "8677b05f-e355-4c69-b16e-8e64d8b0e701") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 13:55:37.221699 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:37.221532    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 13:55:37.221699 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:37.221588    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14057e5c-c9fb-4894-869c-9780abc9c769-original-pull-secret podName:14057e5c-c9fb-4894-869c-9780abc9c769 nodeName:}" failed. No retries permitted until 2026-09-17 13:55:41.22157543 +0000 UTC m=+10.163719058 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/14057e5c-c9fb-4894-869c-9780abc9c769-original-pull-secret") pod "global-pull-secret-syncer-58g8x" (UID: "14057e5c-c9fb-4894-869c-9780abc9c769") : object "kube-system"/"original-pull-secret" not registered
Sep 17 13:55:37.564497 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:37.563712    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9jwwx"
Sep 17 13:55:37.564497 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:37.563831    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9jwwx" podUID="8677b05f-e355-4c69-b16e-8e64d8b0e701"
Sep 17 13:55:37.564497 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:37.564227    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqcvb"
Sep 17 13:55:37.564497 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:37.564332    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqcvb" podUID="d12b5b83-268b-404c-b77b-dd8ec7ea24d1"
Sep 17 13:55:37.565304 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:37.565023    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58g8x"
Sep 17 13:55:37.565304 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:37.565126    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-58g8x" podUID="14057e5c-c9fb-4894-869c-9780abc9c769"
Sep 17 13:55:39.563945 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:39.563915    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9jwwx"
Sep 17 13:55:39.564411 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:39.564051    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9jwwx" podUID="8677b05f-e355-4c69-b16e-8e64d8b0e701"
Sep 17 13:55:39.564411 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:39.564130    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqcvb"
Sep 17 13:55:39.564411 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:39.564237    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqcvb" podUID="d12b5b83-268b-404c-b77b-dd8ec7ea24d1"
Sep 17 13:55:39.564411 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:39.564273    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58g8x"
Sep 17 13:55:39.564411 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:39.564362    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-58g8x" podUID="14057e5c-c9fb-4894-869c-9780abc9c769"
Sep 17 13:55:41.255868 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:41.255461    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rndqv\" (UniqueName: \"kubernetes.io/projected/8677b05f-e355-4c69-b16e-8e64d8b0e701-kube-api-access-rndqv\") pod \"network-check-target-9jwwx\" (UID: \"8677b05f-e355-4c69-b16e-8e64d8b0e701\") " pod="openshift-network-diagnostics/network-check-target-9jwwx"
Sep 17 13:55:41.255868 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:41.255561    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/14057e5c-c9fb-4894-869c-9780abc9c769-original-pull-secret\") pod \"global-pull-secret-syncer-58g8x\" (UID: \"14057e5c-c9fb-4894-869c-9780abc9c769\") " pod="kube-system/global-pull-secret-syncer-58g8x"
Sep 17 13:55:41.255868 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:41.255594    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d12b5b83-268b-404c-b77b-dd8ec7ea24d1-metrics-certs\") pod \"network-metrics-daemon-dqcvb\" (UID: \"d12b5b83-268b-404c-b77b-dd8ec7ea24d1\") " pod="openshift-multus/network-metrics-daemon-dqcvb"
Sep 17 13:55:41.255868 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:41.255732    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 13:55:41.255868 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:41.255793    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d12b5b83-268b-404c-b77b-dd8ec7ea24d1-metrics-certs podName:d12b5b83-268b-404c-b77b-dd8ec7ea24d1 nodeName:}" failed. No retries permitted until 2026-09-17 13:55:49.255773462 +0000 UTC m=+18.197917094 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d12b5b83-268b-404c-b77b-dd8ec7ea24d1-metrics-certs") pod "network-metrics-daemon-dqcvb" (UID: "d12b5b83-268b-404c-b77b-dd8ec7ea24d1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 13:55:41.255868 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:41.255876    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 13:55:41.256648 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:41.255889    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 13:55:41.256648 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:41.255911    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 13:55:41.256648 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:41.255926    2560 projected.go:196] Error preparing data for projected volume kube-api-access-rndqv for pod openshift-network-diagnostics/network-check-target-9jwwx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 13:55:41.256648 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:41.255977    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14057e5c-c9fb-4894-869c-9780abc9c769-original-pull-secret podName:14057e5c-c9fb-4894-869c-9780abc9c769 nodeName:}" failed. No retries permitted until 2026-09-17 13:55:49.255923121 +0000 UTC m=+18.198066753 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/14057e5c-c9fb-4894-869c-9780abc9c769-original-pull-secret") pod "global-pull-secret-syncer-58g8x" (UID: "14057e5c-c9fb-4894-869c-9780abc9c769") : object "kube-system"/"original-pull-secret" not registered
Sep 17 13:55:41.256648 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:41.256000    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8677b05f-e355-4c69-b16e-8e64d8b0e701-kube-api-access-rndqv podName:8677b05f-e355-4c69-b16e-8e64d8b0e701 nodeName:}" failed. No retries permitted until 2026-09-17 13:55:49.255987955 +0000 UTC m=+18.198131588 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-rndqv" (UniqueName: "kubernetes.io/projected/8677b05f-e355-4c69-b16e-8e64d8b0e701-kube-api-access-rndqv") pod "network-check-target-9jwwx" (UID: "8677b05f-e355-4c69-b16e-8e64d8b0e701") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 13:55:41.565922 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:41.565143    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58g8x"
Sep 17 13:55:41.565922 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:41.565270    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-58g8x" podUID="14057e5c-c9fb-4894-869c-9780abc9c769"
Sep 17 13:55:41.566859 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:41.566728    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9jwwx"
Sep 17 13:55:41.566859 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:41.566819    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9jwwx" podUID="8677b05f-e355-4c69-b16e-8e64d8b0e701"
Sep 17 13:55:41.567061 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:41.566887    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqcvb"
Sep 17 13:55:41.567061 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:41.567016    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqcvb" podUID="d12b5b83-268b-404c-b77b-dd8ec7ea24d1"
Sep 17 13:55:43.563196 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:43.562889    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58g8x"
Sep 17 13:55:43.563642 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:43.562896    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9jwwx"
Sep 17 13:55:43.563642 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:43.563310    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-58g8x" podUID="14057e5c-c9fb-4894-869c-9780abc9c769"
Sep 17 13:55:43.563642 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:43.563399    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9jwwx" podUID="8677b05f-e355-4c69-b16e-8e64d8b0e701"
Sep 17 13:55:43.563642 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:43.562936    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqcvb"
Sep 17 13:55:43.563642 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:43.563531    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqcvb" podUID="d12b5b83-268b-404c-b77b-dd8ec7ea24d1"
Sep 17 13:55:45.562946 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:45.562912    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58g8x"
Sep 17 13:55:45.563383 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:45.562918    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9jwwx"
Sep 17 13:55:45.563383 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:45.563038    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-58g8x" podUID="14057e5c-c9fb-4894-869c-9780abc9c769"
Sep 17 13:55:45.563383 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:45.562924    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqcvb"
Sep 17 13:55:45.563383 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:45.563110    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9jwwx" podUID="8677b05f-e355-4c69-b16e-8e64d8b0e701"
Sep 17 13:55:45.563383 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:45.563199    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqcvb" podUID="d12b5b83-268b-404c-b77b-dd8ec7ea24d1"
Sep 17 13:55:47.563663 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:47.563626    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58g8x"
Sep 17 13:55:47.563663 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:47.563643    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9jwwx"
Sep 17 13:55:47.563663 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:47.563663    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqcvb"
Sep 17 13:55:47.564247 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:47.563761    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-58g8x" podUID="14057e5c-c9fb-4894-869c-9780abc9c769"
Sep 17 13:55:47.564247 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:47.563866    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9jwwx" podUID="8677b05f-e355-4c69-b16e-8e64d8b0e701"
Sep 17 13:55:47.564247 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:47.563973    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqcvb" podUID="d12b5b83-268b-404c-b77b-dd8ec7ea24d1"
Sep 17 13:55:49.322395 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:49.322357    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d12b5b83-268b-404c-b77b-dd8ec7ea24d1-metrics-certs\") pod \"network-metrics-daemon-dqcvb\" (UID: \"d12b5b83-268b-404c-b77b-dd8ec7ea24d1\") " pod="openshift-multus/network-metrics-daemon-dqcvb"
Sep 17 13:55:49.322893 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:49.322414    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rndqv\" (UniqueName: \"kubernetes.io/projected/8677b05f-e355-4c69-b16e-8e64d8b0e701-kube-api-access-rndqv\") pod \"network-check-target-9jwwx\" (UID: \"8677b05f-e355-4c69-b16e-8e64d8b0e701\") " pod="openshift-network-diagnostics/network-check-target-9jwwx"
Sep 17 13:55:49.322893 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:49.322492    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/14057e5c-c9fb-4894-869c-9780abc9c769-original-pull-secret\") pod \"global-pull-secret-syncer-58g8x\" (UID: \"14057e5c-c9fb-4894-869c-9780abc9c769\") " pod="kube-system/global-pull-secret-syncer-58g8x"
Sep 17 13:55:49.322893 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:49.322537    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 13:55:49.322893 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:49.322584    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 13:55:49.322893 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:49.322612    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d12b5b83-268b-404c-b77b-dd8ec7ea24d1-metrics-certs podName:d12b5b83-268b-404c-b77b-dd8ec7ea24d1 nodeName:}" failed. No retries permitted until 2026-09-17 13:56:05.322590733 +0000 UTC m=+34.264734361 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d12b5b83-268b-404c-b77b-dd8ec7ea24d1-metrics-certs") pod "network-metrics-daemon-dqcvb" (UID: "d12b5b83-268b-404c-b77b-dd8ec7ea24d1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 13:55:49.322893 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:49.322619    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 13:55:49.322893 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:49.322634    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14057e5c-c9fb-4894-869c-9780abc9c769-original-pull-secret podName:14057e5c-c9fb-4894-869c-9780abc9c769 nodeName:}" failed. No retries permitted until 2026-09-17 13:56:05.322623974 +0000 UTC m=+34.264767606 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/14057e5c-c9fb-4894-869c-9780abc9c769-original-pull-secret") pod "global-pull-secret-syncer-58g8x" (UID: "14057e5c-c9fb-4894-869c-9780abc9c769") : object "kube-system"/"original-pull-secret" not registered
Sep 17 13:55:49.322893 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:49.322641    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 13:55:49.322893 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:49.322654    2560 projected.go:196] Error preparing data for projected volume kube-api-access-rndqv for pod openshift-network-diagnostics/network-check-target-9jwwx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 13:55:49.322893 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:49.322699    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8677b05f-e355-4c69-b16e-8e64d8b0e701-kube-api-access-rndqv podName:8677b05f-e355-4c69-b16e-8e64d8b0e701 nodeName:}" failed. No retries permitted until 2026-09-17 13:56:05.322683139 +0000 UTC m=+34.264826765 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-rndqv" (UniqueName: "kubernetes.io/projected/8677b05f-e355-4c69-b16e-8e64d8b0e701-kube-api-access-rndqv") pod "network-check-target-9jwwx" (UID: "8677b05f-e355-4c69-b16e-8e64d8b0e701") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 13:55:49.562989 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:49.562957    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9jwwx"
Sep 17 13:55:49.562989 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:49.562977    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqcvb"
Sep 17 13:55:49.563211 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:49.563097    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9jwwx" podUID="8677b05f-e355-4c69-b16e-8e64d8b0e701"
Sep 17 13:55:49.563211 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:49.563143    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqcvb" podUID="d12b5b83-268b-404c-b77b-dd8ec7ea24d1"
Sep 17 13:55:49.563311 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:49.563207    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58g8x"
Sep 17 13:55:49.563366 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:49.563310    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-58g8x" podUID="14057e5c-c9fb-4894-869c-9780abc9c769"
Sep 17 13:55:51.563397 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:51.563369    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9jwwx"
Sep 17 13:55:51.563844 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:51.563479    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9jwwx" podUID="8677b05f-e355-4c69-b16e-8e64d8b0e701"
Sep 17 13:55:51.563844 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:51.563572    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqcvb"
Sep 17 13:55:51.563844 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:51.563684    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqcvb" podUID="d12b5b83-268b-404c-b77b-dd8ec7ea24d1"
Sep 17 13:55:51.563844 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:51.563693    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58g8x"
Sep 17 13:55:51.563844 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:51.563773    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-58g8x" podUID="14057e5c-c9fb-4894-869c-9780abc9c769"
Sep 17 13:55:53.567172 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:53.566405    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58g8x"
Sep 17 13:55:53.567172 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:53.566540    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-58g8x" podUID="14057e5c-c9fb-4894-869c-9780abc9c769"
Sep 17 13:55:53.567172 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:53.566911    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9jwwx"
Sep 17 13:55:53.567172 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:53.566999    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9jwwx" podUID="8677b05f-e355-4c69-b16e-8e64d8b0e701"
Sep 17 13:55:53.567172 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:53.567060    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqcvb"
Sep 17 13:55:53.567172 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:53.567133    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqcvb" podUID="d12b5b83-268b-404c-b77b-dd8ec7ea24d1"
Sep 17 13:55:53.738431 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:53.738147    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9wpxw" event={"ID":"9fcb9893-714e-47b5-9b6f-37e706357ffe","Type":"ContainerStarted","Data":"5e1f37568fa05aad4b78dcf2e8a8643225466d6af46c44390e9d6c702db735e6"}
Sep 17 13:55:53.739898 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:53.739876    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2zqvg" event={"ID":"a4a8010a-d12b-4b1a-a0db-e1c64109b151","Type":"ContainerStarted","Data":"bd28df052ee930cdaf25297e74729a09df2a369b90af8c10f83e9284668eeec1"}
Sep 17 13:55:53.741708 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:53.741680    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zxhg5" event={"ID":"bfe7d6ff-377c-4449-8c3c-69163deda18c","Type":"ContainerStarted","Data":"76736c57ba0198b07505132b51f884ec1935b5c9e9d0dae9632d616b74800d0b"}
Sep 17 13:55:53.743410 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:53.743383    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vqb5r" event={"ID":"ece625b2-eada-4304-a7e0-c96a322f289b","Type":"ContainerStarted","Data":"8770ab0b18c2437bd30d00c45cc66768920cad96070ffa9b9aed7a1c9ce8893e"}
Sep 17 13:55:53.744919 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:53.744895    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6gqm" event={"ID":"714b66d5-d63d-4ac3-a097-f338deef9592","Type":"ContainerStarted","Data":"266331e71378448cd4ff72b5b69742d8576627dd361a6234dfbf70a424584fed"}
Sep 17 13:55:53.746376 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:53.746355    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gqsqk" event={"ID":"eae24b6c-3322-4165-96fa-cd61dab923ca","Type":"ContainerStarted","Data":"0bd6e2d134967f6dc49fa316acfc7dace605496a36587d470190e41d993dcaf9"}
Sep 17 13:55:53.748193 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:53.748171    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6546h" event={"ID":"08545f35-761a-4929-9192-254ca6b76569","Type":"ContainerStarted","Data":"df588349fef45a694d0ebfc700d12ef6174f2c4ae560cf9a9ac00e044d8448c3"}
Sep 17 13:55:53.748281 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:53.748249    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6546h" event={"ID":"08545f35-761a-4929-9192-254ca6b76569","Type":"ContainerStarted","Data":"16fc99a30417e15e149ccfe502bc27fc1f1b1191c722720d1fa281b4ca42c114"}
Sep 17 13:55:53.749952 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:53.749929    2560 generic.go:356] "Generic (PLEG): container finished" podID="e2ae1c9d-26b0-47e9-ae74-53904cc9f089" containerID="9ca933408379593b264c5ed2467010c3a2517e17fbd5e04b56399c65bb66c1f2" exitCode=0
Sep 17 13:55:53.750018 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:53.749978    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ssnxx" event={"ID":"e2ae1c9d-26b0-47e9-ae74-53904cc9f089","Type":"ContainerDied","Data":"9ca933408379593b264c5ed2467010c3a2517e17fbd5e04b56399c65bb66c1f2"}
Sep 17 13:55:53.775699 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:53.775642    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-9wpxw" podStartSLOduration=2.163117079 podStartE2EDuration="20.77562133s" totalImagesPullingTime="18.612504251s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 13:55:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 13:55:53.773598849 +0000 UTC m=+22.715742496" watchObservedRunningTime="2026-09-17 13:55:53.77562133 +0000 UTC m=+22.717764978"
Sep 17 13:55:53.791953 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:53.791910    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zxhg5" podStartSLOduration=3.175069056 podStartE2EDuration="21.791894138s" totalImagesPullingTime="18.616825082s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 13:55:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 13:55:53.791520938 +0000 UTC m=+22.733664584" watchObservedRunningTime="2026-09-17 13:55:53.791894138 +0000 UTC m=+22.734037783"
Sep 17 13:55:53.810115 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:53.810072    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-2zqvg" podStartSLOduration=3.19081136 podStartE2EDuration="21.810056151s" totalImagesPullingTime="18.619244791s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 13:55:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 13:55:53.809306941 +0000 UTC m=+22.751450586" watchObservedRunningTime="2026-09-17 13:55:53.810056151 +0000 UTC m=+22.752199796"
Sep 17 13:55:53.827880 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:53.827834    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-vqb5r" podStartSLOduration=3.167832098 podStartE2EDuration="21.827814166s" totalImagesPullingTime="18.659982068s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 13:55:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 13:55:53.827241003 +0000 UTC m=+22.769384650" watchObservedRunningTime="2026-09-17 13:55:53.827814166 +0000 UTC m=+22.769957813"
Sep 17 13:55:53.881919 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:53.881879    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-gqsqk" podStartSLOduration=5.938937023 podStartE2EDuration="20.881865491s" totalImagesPullingTime="14.942928468s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 13:55:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 13:55:53.881670215 +0000 UTC m=+22.823813861" watchObservedRunningTime="2026-09-17 13:55:53.881865491 +0000 UTC m=+22.824009136"
Sep 17 13:55:54.757442 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:54.757407    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6546h" event={"ID":"08545f35-761a-4929-9192-254ca6b76569","Type":"ContainerStarted","Data":"8c3f4c0f514ea6ef2829adf65057793439d9fd7ab80dd0a977b496213f19f813"}
Sep 17 13:55:54.758376 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:54.757545    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6546h" event={"ID":"08545f35-761a-4929-9192-254ca6b76569","Type":"ContainerStarted","Data":"97657a7dead0a24950dfe0030a156d8bf91580b94b00c0ce2301a130a8d626d3"}
Sep 17 13:55:54.758376 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:54.757593    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6546h" event={"ID":"08545f35-761a-4929-9192-254ca6b76569","Type":"ContainerStarted","Data":"521011946a3babe6442c65267cd96b2613090a2a0336aee1a624605038e971af"}
Sep 17 13:55:54.758376 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:54.757634    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6546h" event={"ID":"08545f35-761a-4929-9192-254ca6b76569","Type":"ContainerStarted","Data":"eb4eeec5fd713f3b7a81d48073633987648cad4df96db43973873526d6d62a2b"}
Sep 17 13:55:55.510852 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:55.510826    2560 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 13:55:55.539941 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:55.539827    2560 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T13:55:55.510848903Z","UUID":"e156f69f-989d-43b6-82b1-179960a3613c","Handler":null,"Name":"","Endpoint":""}
Sep 17 13:55:55.541494 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:55.541475    2560 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 13:55:55.541607 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:55.541504    2560 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 13:55:55.562873 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:55.562845    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9jwwx"
Sep 17 13:55:55.562873 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:55.562865    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58g8x"
Sep 17 13:55:55.563074 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:55.562845    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqcvb"
Sep 17 13:55:55.563074 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:55.562971    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9jwwx" podUID="8677b05f-e355-4c69-b16e-8e64d8b0e701"
Sep 17 13:55:55.563178 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:55.563084    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqcvb" podUID="d12b5b83-268b-404c-b77b-dd8ec7ea24d1"
Sep 17 13:55:55.563231 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:55.563172    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-58g8x" podUID="14057e5c-c9fb-4894-869c-9780abc9c769"
Sep 17 13:55:55.761870 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:55.761743    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xrrk6" event={"ID":"4071dbb7-7923-4beb-856a-f03b5c7cfb98","Type":"ContainerStarted","Data":"2cdf005f463eb29395c08e3994a0a39d714a2f7b48a5cddf57e087652205ae56"}
Sep 17 13:55:55.763797 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:55.763769    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6gqm" event={"ID":"714b66d5-d63d-4ac3-a097-f338deef9592","Type":"ContainerStarted","Data":"6cbcb587ea10274dea932c80e2ca29042691fab1bfd020d3195431e66ef8dc1d"}
Sep 17 13:55:55.779910 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:55.779868    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xrrk6" podStartSLOduration=5.172765884 podStartE2EDuration="23.779854059s" totalImagesPullingTime="18.607088175s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 13:55:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 13:55:55.779652115 +0000 UTC m=+24.721795760" watchObservedRunningTime="2026-09-17 13:55:55.779854059 +0000 UTC m=+24.721997736"
Sep 17 13:55:55.852711 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:55.852678    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-gqsqk"
Sep 17 13:55:55.853506 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:55.853482    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-gqsqk"
Sep 17 13:55:56.774421 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:56.774370    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6546h" event={"ID":"08545f35-761a-4929-9192-254ca6b76569","Type":"ContainerStarted","Data":"592e213298f1e0ad54477f4322c61c751d525a9b83e91600905c96baeb70521c"}
Sep 17 13:55:57.563582 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:57.563503    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58g8x"
Sep 17 13:55:57.563736 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:57.563515    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9jwwx"
Sep 17 13:55:57.563736 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:57.563623    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-58g8x" podUID="14057e5c-c9fb-4894-869c-9780abc9c769"
Sep 17 13:55:57.563736 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:57.563698    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9jwwx" podUID="8677b05f-e355-4c69-b16e-8e64d8b0e701"
Sep 17 13:55:57.563736 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:57.563723    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqcvb"
Sep 17 13:55:57.563943 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:57.563828    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqcvb" podUID="d12b5b83-268b-404c-b77b-dd8ec7ea24d1"
Sep 17 13:55:57.778859 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:57.778824    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6gqm" event={"ID":"714b66d5-d63d-4ac3-a097-f338deef9592","Type":"ContainerStarted","Data":"604e7b9b23d7e141e005adec6281775fccc34de364612bb446d73c39711499d2"}
Sep 17 13:55:57.798383 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:57.798328    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6gqm" podStartSLOduration=3.702847729 podStartE2EDuration="25.798311091s" totalImagesPullingTime="22.095463362s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 13:55:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 13:55:57.798007114 +0000 UTC m=+26.740150762" watchObservedRunningTime="2026-09-17 13:55:57.798311091 +0000 UTC m=+26.740454737"
Sep 17 13:55:59.563323 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:59.563096    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqcvb"
Sep 17 13:55:59.563838 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:59.563103    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58g8x"
Sep 17 13:55:59.563838 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:59.563439    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqcvb" podUID="d12b5b83-268b-404c-b77b-dd8ec7ea24d1"
Sep 17 13:55:59.563838 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:59.563160    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9jwwx"
Sep 17 13:55:59.563838 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:59.563489    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-58g8x" podUID="14057e5c-c9fb-4894-869c-9780abc9c769"
Sep 17 13:55:59.563838 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:55:59.563591    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9jwwx" podUID="8677b05f-e355-4c69-b16e-8e64d8b0e701"
Sep 17 13:55:59.786926 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:59.786846    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6546h" event={"ID":"08545f35-761a-4929-9192-254ca6b76569","Type":"ContainerStarted","Data":"16cb212ab8415b50270e0e0f787540add69c88b4f19bc43aa255b5735b901a2f"}
Sep 17 13:55:59.787098 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:59.787076    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6546h"
Sep 17 13:55:59.788807 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:59.788779    2560 generic.go:356] "Generic (PLEG): container finished" podID="e2ae1c9d-26b0-47e9-ae74-53904cc9f089" containerID="7df1cc40d91509534e6fadcfd54c56765ec81f14cf30c4189b46d7a3523aa19f" exitCode=0
Sep 17 13:55:59.788923 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:59.788831    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ssnxx" event={"ID":"e2ae1c9d-26b0-47e9-ae74-53904cc9f089","Type":"ContainerDied","Data":"7df1cc40d91509534e6fadcfd54c56765ec81f14cf30c4189b46d7a3523aa19f"}
Sep 17 13:55:59.801919 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:59.801902    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6546h"
Sep 17 13:55:59.823093 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:55:59.823059    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-6546h" podStartSLOduration=9.170711509 podStartE2EDuration="27.823047802s" totalImagesPullingTime="18.652336293s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 13:55:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 13:55:59.822141467 +0000 UTC m=+28.764285112" watchObservedRunningTime="2026-09-17 13:55:59.823047802 +0000 UTC m=+28.765191447"
Sep 17 13:56:00.725430 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:00.725402    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6546h"
Sep 17 13:56:00.791929 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:00.791902    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6546h"
Sep 17 13:56:00.806632 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:00.806605    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6546h"
Sep 17 13:56:01.102479 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:01.102283    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-58g8x"]
Sep 17 13:56:01.102596 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:01.102584    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58g8x"
Sep 17 13:56:01.102687 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:56:01.102671    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-58g8x" podUID="14057e5c-c9fb-4894-869c-9780abc9c769"
Sep 17 13:56:01.105545 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:01.105526    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9jwwx"]
Sep 17 13:56:01.105620 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:01.105605    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9jwwx"
Sep 17 13:56:01.105686 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:56:01.105670    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9jwwx" podUID="8677b05f-e355-4c69-b16e-8e64d8b0e701"
Sep 17 13:56:01.108022 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:01.108000    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dqcvb"]
Sep 17 13:56:01.108086 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:01.108080    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqcvb"
Sep 17 13:56:01.108171 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:56:01.108153    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqcvb" podUID="d12b5b83-268b-404c-b77b-dd8ec7ea24d1"
Sep 17 13:56:01.795615 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:01.795584    2560 generic.go:356] "Generic (PLEG): container finished" podID="e2ae1c9d-26b0-47e9-ae74-53904cc9f089" containerID="0ef79dece466005cd37215f15b014e48d6a528daa6696037907950ecd196c015" exitCode=0
Sep 17 13:56:01.796041 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:01.795642    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ssnxx" event={"ID":"e2ae1c9d-26b0-47e9-ae74-53904cc9f089","Type":"ContainerDied","Data":"0ef79dece466005cd37215f15b014e48d6a528daa6696037907950ecd196c015"}
Sep 17 13:56:02.563477 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:02.563444    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9jwwx"
Sep 17 13:56:02.563635 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:02.563480    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58g8x"
Sep 17 13:56:02.563635 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:02.563444    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqcvb"
Sep 17 13:56:02.563635 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:56:02.563584    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9jwwx" podUID="8677b05f-e355-4c69-b16e-8e64d8b0e701"
Sep 17 13:56:02.563760 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:56:02.563660    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqcvb" podUID="d12b5b83-268b-404c-b77b-dd8ec7ea24d1"
Sep 17 13:56:02.563792 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:56:02.563751    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-58g8x" podUID="14057e5c-c9fb-4894-869c-9780abc9c769"
Sep 17 13:56:03.465379 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:03.465307    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-gqsqk"
Sep 17 13:56:03.465797 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:03.465446    2560 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 13:56:03.465935 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:03.465912    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-gqsqk"
Sep 17 13:56:03.802010 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:03.801978    2560 generic.go:356] "Generic (PLEG): container finished" podID="e2ae1c9d-26b0-47e9-ae74-53904cc9f089" containerID="d7d7e6e5d83f2d8db4685d7c480e7f80b5a460a4901ace367084ac671c08d4b1" exitCode=0
Sep 17 13:56:03.802131 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:03.802051    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ssnxx" event={"ID":"e2ae1c9d-26b0-47e9-ae74-53904cc9f089","Type":"ContainerDied","Data":"d7d7e6e5d83f2d8db4685d7c480e7f80b5a460a4901ace367084ac671c08d4b1"}
Sep 17 13:56:04.562861 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:04.562834    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqcvb"
Sep 17 13:56:04.563195 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:04.562834    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9jwwx"
Sep 17 13:56:04.563195 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:56:04.562932    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqcvb" podUID="d12b5b83-268b-404c-b77b-dd8ec7ea24d1"
Sep 17 13:56:04.563195 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:04.562834    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58g8x"
Sep 17 13:56:04.563195 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:56:04.562995    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9jwwx" podUID="8677b05f-e355-4c69-b16e-8e64d8b0e701"
Sep 17 13:56:04.563195 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:56:04.563064    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-58g8x" podUID="14057e5c-c9fb-4894-869c-9780abc9c769"
Sep 17 13:56:05.345430 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:05.345397    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d12b5b83-268b-404c-b77b-dd8ec7ea24d1-metrics-certs\") pod \"network-metrics-daemon-dqcvb\" (UID: \"d12b5b83-268b-404c-b77b-dd8ec7ea24d1\") " pod="openshift-multus/network-metrics-daemon-dqcvb"
Sep 17 13:56:05.345673 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:05.345447    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rndqv\" (UniqueName: \"kubernetes.io/projected/8677b05f-e355-4c69-b16e-8e64d8b0e701-kube-api-access-rndqv\") pod \"network-check-target-9jwwx\" (UID: \"8677b05f-e355-4c69-b16e-8e64d8b0e701\") " pod="openshift-network-diagnostics/network-check-target-9jwwx"
Sep 17 13:56:05.345673 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:05.345497    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/14057e5c-c9fb-4894-869c-9780abc9c769-original-pull-secret\") pod \"global-pull-secret-syncer-58g8x\" (UID: \"14057e5c-c9fb-4894-869c-9780abc9c769\") " pod="kube-system/global-pull-secret-syncer-58g8x"
Sep 17 13:56:05.345673 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:56:05.345563    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 13:56:05.345673 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:56:05.345628    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d12b5b83-268b-404c-b77b-dd8ec7ea24d1-metrics-certs podName:d12b5b83-268b-404c-b77b-dd8ec7ea24d1 nodeName:}" failed. No retries permitted until 2026-09-17 13:56:37.345608666 +0000 UTC m=+66.287752296 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d12b5b83-268b-404c-b77b-dd8ec7ea24d1-metrics-certs") pod "network-metrics-daemon-dqcvb" (UID: "d12b5b83-268b-404c-b77b-dd8ec7ea24d1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 13:56:05.345673 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:56:05.345641    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 13:56:05.345673 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:56:05.345652    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 13:56:05.345673 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:56:05.345664    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 13:56:05.345673 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:56:05.345678    2560 projected.go:196] Error preparing data for projected volume kube-api-access-rndqv for pod openshift-network-diagnostics/network-check-target-9jwwx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 13:56:05.346203 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:56:05.345701    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14057e5c-c9fb-4894-869c-9780abc9c769-original-pull-secret podName:14057e5c-c9fb-4894-869c-9780abc9c769 nodeName:}" failed. No retries permitted until 2026-09-17 13:56:37.345689871 +0000 UTC m=+66.287833495 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/14057e5c-c9fb-4894-869c-9780abc9c769-original-pull-secret") pod "global-pull-secret-syncer-58g8x" (UID: "14057e5c-c9fb-4894-869c-9780abc9c769") : object "kube-system"/"original-pull-secret" not registered
Sep 17 13:56:05.346203 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:56:05.345722    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8677b05f-e355-4c69-b16e-8e64d8b0e701-kube-api-access-rndqv podName:8677b05f-e355-4c69-b16e-8e64d8b0e701 nodeName:}" failed. No retries permitted until 2026-09-17 13:56:37.345707323 +0000 UTC m=+66.287850956 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-rndqv" (UniqueName: "kubernetes.io/projected/8677b05f-e355-4c69-b16e-8e64d8b0e701-kube-api-access-rndqv") pod "network-check-target-9jwwx" (UID: "8677b05f-e355-4c69-b16e-8e64d8b0e701") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 13:56:06.348563 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.348535    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-158.ec2.internal" event="NodeReady"
Sep 17 13:56:06.348994 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.348714    2560 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 13:56:06.383824 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.383789    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-d47ffd9b-tr5bs"]
Sep 17 13:56:06.387225 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.387207    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d47ffd9b-tr5bs"
Sep 17 13:56:06.389357 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.389335    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 13:56:06.389530 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.389509    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 13:56:06.389530 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.389522    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-cjxbg\""
Sep 17 13:56:06.389677 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.389569    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 13:56:06.401333 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.401308    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-82dhq"]
Sep 17 13:56:06.402053 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.402030    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 13:56:06.405184 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.405165    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-d47ffd9b-tr5bs"]
Sep 17 13:56:06.405294 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.405197    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-wwqnd"]
Sep 17 13:56:06.405448 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.405432    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-82dhq"
Sep 17 13:56:06.407759 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.407741    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 13:56:06.407867 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.407848    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-f8wrh\""
Sep 17 13:56:06.408246 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.408226    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 13:56:06.409046 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.409027    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wwqnd"
Sep 17 13:56:06.410852 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.410827    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-wntfx\""
Sep 17 13:56:06.411730 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.411660    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 13:56:06.411730 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.411693    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 13:56:06.411887 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.411829    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 13:56:06.419269 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.419248    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wwqnd"]
Sep 17 13:56:06.421848 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.421825    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-82dhq"]
Sep 17 13:56:06.553227 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.553197    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fb692cc1-e158-4666-baca-0d719d8c8aa6-image-registry-private-configuration\") pod \"image-registry-d47ffd9b-tr5bs\" (UID: \"fb692cc1-e158-4666-baca-0d719d8c8aa6\") " pod="openshift-image-registry/image-registry-d47ffd9b-tr5bs"
Sep 17 13:56:06.553389 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.553238    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/45035743-85eb-4642-a5c9-adea5c5db4e1-config-volume\") pod \"dns-default-82dhq\" (UID: \"45035743-85eb-4642-a5c9-adea5c5db4e1\") " pod="openshift-dns/dns-default-82dhq"
Sep 17 13:56:06.553389 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.553298    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2769cbbe-3a80-48c0-bf6b-a993360d1ca7-cert\") pod \"ingress-canary-wwqnd\" (UID: \"2769cbbe-3a80-48c0-bf6b-a993360d1ca7\") " pod="openshift-ingress-canary/ingress-canary-wwqnd"
Sep 17 13:56:06.553389 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.553322    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45035743-85eb-4642-a5c9-adea5c5db4e1-tmp-dir\") pod \"dns-default-82dhq\" (UID: \"45035743-85eb-4642-a5c9-adea5c5db4e1\") " pod="openshift-dns/dns-default-82dhq"
Sep 17 13:56:06.553389 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.553349    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m28rd\" (UniqueName: \"kubernetes.io/projected/45035743-85eb-4642-a5c9-adea5c5db4e1-kube-api-access-m28rd\") pod \"dns-default-82dhq\" (UID: \"45035743-85eb-4642-a5c9-adea5c5db4e1\") " pod="openshift-dns/dns-default-82dhq"
Sep 17 13:56:06.553389 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.553378    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fb692cc1-e158-4666-baca-0d719d8c8aa6-installation-pull-secrets\") pod \"image-registry-d47ffd9b-tr5bs\" (UID: \"fb692cc1-e158-4666-baca-0d719d8c8aa6\") " pod="openshift-image-registry/image-registry-d47ffd9b-tr5bs"
Sep 17 13:56:06.553590 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.553446    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/45035743-85eb-4642-a5c9-adea5c5db4e1-metrics-tls\") pod \"dns-default-82dhq\" (UID: \"45035743-85eb-4642-a5c9-adea5c5db4e1\") " pod="openshift-dns/dns-default-82dhq"
Sep 17 13:56:06.553590 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.553506    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fb692cc1-e158-4666-baca-0d719d8c8aa6-ca-trust-extracted\") pod \"image-registry-d47ffd9b-tr5bs\" (UID: \"fb692cc1-e158-4666-baca-0d719d8c8aa6\") " pod="openshift-image-registry/image-registry-d47ffd9b-tr5bs"
Sep 17 13:56:06.553590 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.553535    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hhhr\" (UniqueName: \"kubernetes.io/projected/fb692cc1-e158-4666-baca-0d719d8c8aa6-kube-api-access-2hhhr\") pod \"image-registry-d47ffd9b-tr5bs\" (UID: \"fb692cc1-e158-4666-baca-0d719d8c8aa6\") " pod="openshift-image-registry/image-registry-d47ffd9b-tr5bs"
Sep 17 13:56:06.553590 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.553583    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fb692cc1-e158-4666-baca-0d719d8c8aa6-registry-certificates\") pod \"image-registry-d47ffd9b-tr5bs\" (UID: \"fb692cc1-e158-4666-baca-0d719d8c8aa6\") " pod="openshift-image-registry/image-registry-d47ffd9b-tr5bs"
Sep 17 13:56:06.553736 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.553609    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fb692cc1-e158-4666-baca-0d719d8c8aa6-trusted-ca\") pod \"image-registry-d47ffd9b-tr5bs\" (UID: \"fb692cc1-e158-4666-baca-0d719d8c8aa6\") " pod="openshift-image-registry/image-registry-d47ffd9b-tr5bs"
Sep 17 13:56:06.553736 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.553658    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hhdh\" (UniqueName: \"kubernetes.io/projected/2769cbbe-3a80-48c0-bf6b-a993360d1ca7-kube-api-access-5hhdh\") pod \"ingress-canary-wwqnd\" (UID: \"2769cbbe-3a80-48c0-bf6b-a993360d1ca7\") " pod="openshift-ingress-canary/ingress-canary-wwqnd"
Sep 17 13:56:06.553736 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.553684    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fb692cc1-e158-4666-baca-0d719d8c8aa6-registry-tls\") pod \"image-registry-d47ffd9b-tr5bs\" (UID: \"fb692cc1-e158-4666-baca-0d719d8c8aa6\") " pod="openshift-image-registry/image-registry-d47ffd9b-tr5bs"
Sep 17 13:56:06.553849 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.553764    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fb692cc1-e158-4666-baca-0d719d8c8aa6-bound-sa-token\") pod \"image-registry-d47ffd9b-tr5bs\" (UID: \"fb692cc1-e158-4666-baca-0d719d8c8aa6\") " pod="openshift-image-registry/image-registry-d47ffd9b-tr5bs"
Sep 17 13:56:06.563235 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.563213    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9jwwx"
Sep 17 13:56:06.563364 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.563213    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58g8x"
Sep 17 13:56:06.563427 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.563213    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqcvb"
Sep 17 13:56:06.565650 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.565617    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 13:56:06.565650 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.565625    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gs47f\""
Sep 17 13:56:06.565823 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.565683    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 13:56:06.565823 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.565683    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 13:56:06.565823 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.565617    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-cpdmf\""
Sep 17 13:56:06.565823 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.565729    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 13:56:06.654115 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.654085    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fb692cc1-e158-4666-baca-0d719d8c8aa6-image-registry-private-configuration\") pod \"image-registry-d47ffd9b-tr5bs\" (UID: \"fb692cc1-e158-4666-baca-0d719d8c8aa6\") " pod="openshift-image-registry/image-registry-d47ffd9b-tr5bs"
Sep 17 13:56:06.654293 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.654127    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/45035743-85eb-4642-a5c9-adea5c5db4e1-config-volume\") pod \"dns-default-82dhq\" (UID: \"45035743-85eb-4642-a5c9-adea5c5db4e1\") " pod="openshift-dns/dns-default-82dhq"
Sep 17 13:56:06.654293 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.654162    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2769cbbe-3a80-48c0-bf6b-a993360d1ca7-cert\") pod \"ingress-canary-wwqnd\" (UID: \"2769cbbe-3a80-48c0-bf6b-a993360d1ca7\") " pod="openshift-ingress-canary/ingress-canary-wwqnd"
Sep 17 13:56:06.654293 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.654188    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45035743-85eb-4642-a5c9-adea5c5db4e1-tmp-dir\") pod \"dns-default-82dhq\" (UID: \"45035743-85eb-4642-a5c9-adea5c5db4e1\") " pod="openshift-dns/dns-default-82dhq"
Sep 17 13:56:06.654293 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.654214    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m28rd\" (UniqueName: \"kubernetes.io/projected/45035743-85eb-4642-a5c9-adea5c5db4e1-kube-api-access-m28rd\") pod \"dns-default-82dhq\" (UID: \"45035743-85eb-4642-a5c9-adea5c5db4e1\") " pod="openshift-dns/dns-default-82dhq"
Sep 17 13:56:06.654293 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.654242    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fb692cc1-e158-4666-baca-0d719d8c8aa6-installation-pull-secrets\") pod \"image-registry-d47ffd9b-tr5bs\" (UID: \"fb692cc1-e158-4666-baca-0d719d8c8aa6\") " pod="openshift-image-registry/image-registry-d47ffd9b-tr5bs"
Sep 17 13:56:06.654567 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.654299    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/45035743-85eb-4642-a5c9-adea5c5db4e1-metrics-tls\") pod \"dns-default-82dhq\" (UID: \"45035743-85eb-4642-a5c9-adea5c5db4e1\") " pod="openshift-dns/dns-default-82dhq"
Sep 17 13:56:06.654567 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.654326    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fb692cc1-e158-4666-baca-0d719d8c8aa6-ca-trust-extracted\") pod \"image-registry-d47ffd9b-tr5bs\" (UID: \"fb692cc1-e158-4666-baca-0d719d8c8aa6\") " pod="openshift-image-registry/image-registry-d47ffd9b-tr5bs"
Sep 17 13:56:06.654567 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.654355    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hhhr\" (UniqueName: \"kubernetes.io/projected/fb692cc1-e158-4666-baca-0d719d8c8aa6-kube-api-access-2hhhr\") pod \"image-registry-d47ffd9b-tr5bs\" (UID: \"fb692cc1-e158-4666-baca-0d719d8c8aa6\") " pod="openshift-image-registry/image-registry-d47ffd9b-tr5bs"
Sep 17 13:56:06.654567 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.654398    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fb692cc1-e158-4666-baca-0d719d8c8aa6-registry-certificates\") pod \"image-registry-d47ffd9b-tr5bs\" (UID: \"fb692cc1-e158-4666-baca-0d719d8c8aa6\") " pod="openshift-image-registry/image-registry-d47ffd9b-tr5bs"
Sep 17 13:56:06.654567 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.654422    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fb692cc1-e158-4666-baca-0d719d8c8aa6-trusted-ca\") pod \"image-registry-d47ffd9b-tr5bs\" (UID: \"fb692cc1-e158-4666-baca-0d719d8c8aa6\") " pod="openshift-image-registry/image-registry-d47ffd9b-tr5bs"
Sep 17 13:56:06.654567 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.654477    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hhdh\" (UniqueName: \"kubernetes.io/projected/2769cbbe-3a80-48c0-bf6b-a993360d1ca7-kube-api-access-5hhdh\") pod \"ingress-canary-wwqnd\" (UID: \"2769cbbe-3a80-48c0-bf6b-a993360d1ca7\") " pod="openshift-ingress-canary/ingress-canary-wwqnd"
Sep 17 13:56:06.654567 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.654501    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fb692cc1-e158-4666-baca-0d719d8c8aa6-registry-tls\") pod \"image-registry-d47ffd9b-tr5bs\" (UID: \"fb692cc1-e158-4666-baca-0d719d8c8aa6\") " pod="openshift-image-registry/image-registry-d47ffd9b-tr5bs"
Sep 17 13:56:06.654567 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.654527    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fb692cc1-e158-4666-baca-0d719d8c8aa6-bound-sa-token\") pod \"image-registry-d47ffd9b-tr5bs\" (UID: \"fb692cc1-e158-4666-baca-0d719d8c8aa6\") " pod="openshift-image-registry/image-registry-d47ffd9b-tr5bs"
Sep 17 13:56:06.654950 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.654777    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/45035743-85eb-4642-a5c9-adea5c5db4e1-config-volume\") pod \"dns-default-82dhq\" (UID: \"45035743-85eb-4642-a5c9-adea5c5db4e1\") " pod="openshift-dns/dns-default-82dhq"
Sep 17 13:56:06.655785 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.655123    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fb692cc1-e158-4666-baca-0d719d8c8aa6-ca-trust-extracted\") pod \"image-registry-d47ffd9b-tr5bs\" (UID: \"fb692cc1-e158-4666-baca-0d719d8c8aa6\") " pod="openshift-image-registry/image-registry-d47ffd9b-tr5bs"
Sep 17 13:56:06.655785 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:56:06.655380    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 13:56:06.655785 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.655386    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45035743-85eb-4642-a5c9-adea5c5db4e1-tmp-dir\") pod \"dns-default-82dhq\" (UID: \"45035743-85eb-4642-a5c9-adea5c5db4e1\") " pod="openshift-dns/dns-default-82dhq"
Sep 17 13:56:06.655785 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:56:06.655400    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d47ffd9b-tr5bs: secret "image-registry-tls" not found
Sep 17 13:56:06.655785 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:56:06.655495    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb692cc1-e158-4666-baca-0d719d8c8aa6-registry-tls podName:fb692cc1-e158-4666-baca-0d719d8c8aa6 nodeName:}" failed. No retries permitted until 2026-09-17 13:56:07.155472918 +0000 UTC m=+36.097616557 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fb692cc1-e158-4666-baca-0d719d8c8aa6-registry-tls") pod "image-registry-d47ffd9b-tr5bs" (UID: "fb692cc1-e158-4666-baca-0d719d8c8aa6") : secret "image-registry-tls" not found
Sep 17 13:56:06.655785 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.655611    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fb692cc1-e158-4666-baca-0d719d8c8aa6-registry-certificates\") pod \"image-registry-d47ffd9b-tr5bs\" (UID: \"fb692cc1-e158-4666-baca-0d719d8c8aa6\") " pod="openshift-image-registry/image-registry-d47ffd9b-tr5bs"
Sep 17 13:56:06.655785 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:56:06.655706    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 13:56:06.655785 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:56:06.655761    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45035743-85eb-4642-a5c9-adea5c5db4e1-metrics-tls podName:45035743-85eb-4642-a5c9-adea5c5db4e1 nodeName:}" failed. No retries permitted until 2026-09-17 13:56:07.15574531 +0000 UTC m=+36.097888948 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/45035743-85eb-4642-a5c9-adea5c5db4e1-metrics-tls") pod "dns-default-82dhq" (UID: "45035743-85eb-4642-a5c9-adea5c5db4e1") : secret "dns-default-metrics-tls" not found
Sep 17 13:56:06.656440 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.656416    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fb692cc1-e158-4666-baca-0d719d8c8aa6-trusted-ca\") pod \"image-registry-d47ffd9b-tr5bs\" (UID: \"fb692cc1-e158-4666-baca-0d719d8c8aa6\") " pod="openshift-image-registry/image-registry-d47ffd9b-tr5bs"
Sep 17 13:56:06.659779 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.659645    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fb692cc1-e158-4666-baca-0d719d8c8aa6-installation-pull-secrets\") pod \"image-registry-d47ffd9b-tr5bs\" (UID: \"fb692cc1-e158-4666-baca-0d719d8c8aa6\") " pod="openshift-image-registry/image-registry-d47ffd9b-tr5bs"
Sep 17 13:56:06.659863 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.659651    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fb692cc1-e158-4666-baca-0d719d8c8aa6-image-registry-private-configuration\") pod \"image-registry-d47ffd9b-tr5bs\" (UID: \"fb692cc1-e158-4666-baca-0d719d8c8aa6\") " pod="openshift-image-registry/image-registry-d47ffd9b-tr5bs"
Sep 17 13:56:06.659863 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.659668    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2769cbbe-3a80-48c0-bf6b-a993360d1ca7-cert\") pod \"ingress-canary-wwqnd\" (UID: \"2769cbbe-3a80-48c0-bf6b-a993360d1ca7\") " pod="openshift-ingress-canary/ingress-canary-wwqnd"
Sep 17 13:56:06.664267 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.664243    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hhdh\" (UniqueName: \"kubernetes.io/projected/2769cbbe-3a80-48c0-bf6b-a993360d1ca7-kube-api-access-5hhdh\") pod \"ingress-canary-wwqnd\" (UID: \"2769cbbe-3a80-48c0-bf6b-a993360d1ca7\") " pod="openshift-ingress-canary/ingress-canary-wwqnd"
Sep 17 13:56:06.664842 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.664820    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m28rd\" (UniqueName: \"kubernetes.io/projected/45035743-85eb-4642-a5c9-adea5c5db4e1-kube-api-access-m28rd\") pod \"dns-default-82dhq\" (UID: \"45035743-85eb-4642-a5c9-adea5c5db4e1\") " pod="openshift-dns/dns-default-82dhq"
Sep 17 13:56:06.664947 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.664823    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fb692cc1-e158-4666-baca-0d719d8c8aa6-bound-sa-token\") pod \"image-registry-d47ffd9b-tr5bs\" (UID: \"fb692cc1-e158-4666-baca-0d719d8c8aa6\") " pod="openshift-image-registry/image-registry-d47ffd9b-tr5bs"
Sep 17 13:56:06.665089 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.665073    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hhhr\" (UniqueName: \"kubernetes.io/projected/fb692cc1-e158-4666-baca-0d719d8c8aa6-kube-api-access-2hhhr\") pod \"image-registry-d47ffd9b-tr5bs\" (UID: \"fb692cc1-e158-4666-baca-0d719d8c8aa6\") " pod="openshift-image-registry/image-registry-d47ffd9b-tr5bs"
Sep 17 13:56:06.724328 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:06.724299    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wwqnd"
Sep 17 13:56:07.158997 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:07.158956    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/45035743-85eb-4642-a5c9-adea5c5db4e1-metrics-tls\") pod \"dns-default-82dhq\" (UID: \"45035743-85eb-4642-a5c9-adea5c5db4e1\") " pod="openshift-dns/dns-default-82dhq"
Sep 17 13:56:07.159172 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:07.159027    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fb692cc1-e158-4666-baca-0d719d8c8aa6-registry-tls\") pod \"image-registry-d47ffd9b-tr5bs\" (UID: \"fb692cc1-e158-4666-baca-0d719d8c8aa6\") " pod="openshift-image-registry/image-registry-d47ffd9b-tr5bs"
Sep 17 13:56:07.159172 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:56:07.159080    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 13:56:07.159172 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:56:07.159111    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 13:56:07.159172 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:56:07.159122    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d47ffd9b-tr5bs: secret "image-registry-tls" not found
Sep 17 13:56:07.159172 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:56:07.159159    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45035743-85eb-4642-a5c9-adea5c5db4e1-metrics-tls podName:45035743-85eb-4642-a5c9-adea5c5db4e1 nodeName:}" failed. No retries permitted until 2026-09-17 13:56:08.159139911 +0000 UTC m=+37.101283551 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/45035743-85eb-4642-a5c9-adea5c5db4e1-metrics-tls") pod "dns-default-82dhq" (UID: "45035743-85eb-4642-a5c9-adea5c5db4e1") : secret "dns-default-metrics-tls" not found
Sep 17 13:56:07.159439 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:56:07.159178    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb692cc1-e158-4666-baca-0d719d8c8aa6-registry-tls podName:fb692cc1-e158-4666-baca-0d719d8c8aa6 nodeName:}" failed. No retries permitted until 2026-09-17 13:56:08.159168803 +0000 UTC m=+37.101312426 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fb692cc1-e158-4666-baca-0d719d8c8aa6-registry-tls") pod "image-registry-d47ffd9b-tr5bs" (UID: "fb692cc1-e158-4666-baca-0d719d8c8aa6") : secret "image-registry-tls" not found
Sep 17 13:56:08.166361 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:08.166331    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fb692cc1-e158-4666-baca-0d719d8c8aa6-registry-tls\") pod \"image-registry-d47ffd9b-tr5bs\" (UID: \"fb692cc1-e158-4666-baca-0d719d8c8aa6\") " pod="openshift-image-registry/image-registry-d47ffd9b-tr5bs"
Sep 17 13:56:08.166843 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:08.166416    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/45035743-85eb-4642-a5c9-adea5c5db4e1-metrics-tls\") pod \"dns-default-82dhq\" (UID: \"45035743-85eb-4642-a5c9-adea5c5db4e1\") " pod="openshift-dns/dns-default-82dhq"
Sep 17 13:56:08.166843 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:56:08.166516    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 13:56:08.166843 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:56:08.166516    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 13:56:08.166843 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:56:08.166540    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d47ffd9b-tr5bs: secret "image-registry-tls" not found
Sep 17 13:56:08.166843 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:56:08.166573    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45035743-85eb-4642-a5c9-adea5c5db4e1-metrics-tls podName:45035743-85eb-4642-a5c9-adea5c5db4e1 nodeName:}" failed. No retries permitted until 2026-09-17 13:56:10.166560649 +0000 UTC m=+39.108704286 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/45035743-85eb-4642-a5c9-adea5c5db4e1-metrics-tls") pod "dns-default-82dhq" (UID: "45035743-85eb-4642-a5c9-adea5c5db4e1") : secret "dns-default-metrics-tls" not found
Sep 17 13:56:08.166843 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:56:08.166615    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb692cc1-e158-4666-baca-0d719d8c8aa6-registry-tls podName:fb692cc1-e158-4666-baca-0d719d8c8aa6 nodeName:}" failed. No retries permitted until 2026-09-17 13:56:10.166595779 +0000 UTC m=+39.108739408 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fb692cc1-e158-4666-baca-0d719d8c8aa6-registry-tls") pod "image-registry-d47ffd9b-tr5bs" (UID: "fb692cc1-e158-4666-baca-0d719d8c8aa6") : secret "image-registry-tls" not found
Sep 17 13:56:08.328000 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:08.327970    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wwqnd"]
Sep 17 13:56:08.331813 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:56:08.331783    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2769cbbe_3a80_48c0_bf6b_a993360d1ca7.slice/crio-270a30196fa89bee3b50650b5097cd8b558ef9a4d5f4f7592532e6810c74b55e WatchSource:0}: Error finding container 270a30196fa89bee3b50650b5097cd8b558ef9a4d5f4f7592532e6810c74b55e: Status 404 returned error can't find the container with id 270a30196fa89bee3b50650b5097cd8b558ef9a4d5f4f7592532e6810c74b55e
Sep 17 13:56:08.817701 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:08.817668    2560 generic.go:356] "Generic (PLEG): container finished" podID="e2ae1c9d-26b0-47e9-ae74-53904cc9f089" containerID="3682683b729a8d0f01a7ffa989ecec5833eaa6d9f0f04edba1ed876e8dd35142" exitCode=0
Sep 17 13:56:08.817874 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:08.817762    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ssnxx" event={"ID":"e2ae1c9d-26b0-47e9-ae74-53904cc9f089","Type":"ContainerDied","Data":"3682683b729a8d0f01a7ffa989ecec5833eaa6d9f0f04edba1ed876e8dd35142"}
Sep 17 13:56:08.818779 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:08.818759    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wwqnd" event={"ID":"2769cbbe-3a80-48c0-bf6b-a993360d1ca7","Type":"ContainerStarted","Data":"270a30196fa89bee3b50650b5097cd8b558ef9a4d5f4f7592532e6810c74b55e"}
Sep 17 13:56:09.825165 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:09.825128    2560 generic.go:356] "Generic (PLEG): container finished" podID="e2ae1c9d-26b0-47e9-ae74-53904cc9f089" containerID="069be7ae8cb1b121f4f69d778d1df5298e3517cbb8ef84c70bb6250d44d9a309" exitCode=0
Sep 17 13:56:09.825165 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:09.825170    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ssnxx" event={"ID":"e2ae1c9d-26b0-47e9-ae74-53904cc9f089","Type":"ContainerDied","Data":"069be7ae8cb1b121f4f69d778d1df5298e3517cbb8ef84c70bb6250d44d9a309"}
Sep 17 13:56:10.183531 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:10.183235    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/45035743-85eb-4642-a5c9-adea5c5db4e1-metrics-tls\") pod \"dns-default-82dhq\" (UID: \"45035743-85eb-4642-a5c9-adea5c5db4e1\") " pod="openshift-dns/dns-default-82dhq"
Sep 17 13:56:10.183531 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:10.183508    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fb692cc1-e158-4666-baca-0d719d8c8aa6-registry-tls\") pod \"image-registry-d47ffd9b-tr5bs\" (UID: \"fb692cc1-e158-4666-baca-0d719d8c8aa6\") " pod="openshift-image-registry/image-registry-d47ffd9b-tr5bs"
Sep 17 13:56:10.183531 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:56:10.183392    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 13:56:10.183804 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:56:10.183587    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45035743-85eb-4642-a5c9-adea5c5db4e1-metrics-tls podName:45035743-85eb-4642-a5c9-adea5c5db4e1 nodeName:}" failed. No retries permitted until 2026-09-17 13:56:14.18356727 +0000 UTC m=+43.125710893 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/45035743-85eb-4642-a5c9-adea5c5db4e1-metrics-tls") pod "dns-default-82dhq" (UID: "45035743-85eb-4642-a5c9-adea5c5db4e1") : secret "dns-default-metrics-tls" not found
Sep 17 13:56:10.183804 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:56:10.183632    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 13:56:10.183804 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:56:10.183647    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d47ffd9b-tr5bs: secret "image-registry-tls" not found
Sep 17 13:56:10.183804 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:56:10.183697    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb692cc1-e158-4666-baca-0d719d8c8aa6-registry-tls podName:fb692cc1-e158-4666-baca-0d719d8c8aa6 nodeName:}" failed. No retries permitted until 2026-09-17 13:56:14.18368159 +0000 UTC m=+43.125825220 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fb692cc1-e158-4666-baca-0d719d8c8aa6-registry-tls") pod "image-registry-d47ffd9b-tr5bs" (UID: "fb692cc1-e158-4666-baca-0d719d8c8aa6") : secret "image-registry-tls" not found
Sep 17 13:56:10.646904 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:10.646865    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:56:10.784406 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:10.784364    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:56:10.831856 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:10.831820    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ssnxx" event={"ID":"e2ae1c9d-26b0-47e9-ae74-53904cc9f089","Type":"ContainerStarted","Data":"116cf87bd2e6105f04151d7571ea3a07a0bc854a3ab5d5f929fe6f7e58341a0d"}
Sep 17 13:56:10.862223 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:10.862169    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-ssnxx" podStartSLOduration=8.204374772 podStartE2EDuration="38.862149859s" totalImagesPullingTime="30.657775087s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 13:55:32 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 13:56:10.861542154 +0000 UTC m=+39.803685799" watchObservedRunningTime="2026-09-17 13:56:10.862149859 +0000 UTC m=+39.804293504"
Sep 17 13:56:11.024588 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:11.024495    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:56:11.119807 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:11.119739    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:56:11.680331 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:11.680301    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:56:11.835143 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:11.835068    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wwqnd" event={"ID":"2769cbbe-3a80-48c0-bf6b-a993360d1ca7","Type":"ContainerStarted","Data":"699577b47aa15eb8ad5bed2731475ac40d7f2ad5e1f94e1fa5dbdc5e91a46d81"}
Sep 17 13:56:11.854449 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:11.854407    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-wwqnd" podStartSLOduration=2.61818505 podStartE2EDuration="5.854393342s" totalImagesPullingTime="3.236208292s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 13:56:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 13:56:11.853739889 +0000 UTC m=+40.795883535" watchObservedRunningTime="2026-09-17 13:56:11.854393342 +0000 UTC m=+40.796536989"
Sep 17 13:56:14.216348 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:14.216312    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fb692cc1-e158-4666-baca-0d719d8c8aa6-registry-tls\") pod \"image-registry-d47ffd9b-tr5bs\" (UID: \"fb692cc1-e158-4666-baca-0d719d8c8aa6\") " pod="openshift-image-registry/image-registry-d47ffd9b-tr5bs"
Sep 17 13:56:14.216739 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:14.216383    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/45035743-85eb-4642-a5c9-adea5c5db4e1-metrics-tls\") pod \"dns-default-82dhq\" (UID: \"45035743-85eb-4642-a5c9-adea5c5db4e1\") " pod="openshift-dns/dns-default-82dhq"
Sep 17 13:56:14.216739 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:56:14.216483    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 13:56:14.216739 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:56:14.216503    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d47ffd9b-tr5bs: secret "image-registry-tls" not found
Sep 17 13:56:14.216739 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:56:14.216532    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 13:56:14.216739 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:56:14.216555    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb692cc1-e158-4666-baca-0d719d8c8aa6-registry-tls podName:fb692cc1-e158-4666-baca-0d719d8c8aa6 nodeName:}" failed. No retries permitted until 2026-09-17 13:56:22.216540694 +0000 UTC m=+51.158684318 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fb692cc1-e158-4666-baca-0d719d8c8aa6-registry-tls") pod "image-registry-d47ffd9b-tr5bs" (UID: "fb692cc1-e158-4666-baca-0d719d8c8aa6") : secret "image-registry-tls" not found
Sep 17 13:56:14.216739 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:56:14.216576    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45035743-85eb-4642-a5c9-adea5c5db4e1-metrics-tls podName:45035743-85eb-4642-a5c9-adea5c5db4e1 nodeName:}" failed. No retries permitted until 2026-09-17 13:56:22.21656245 +0000 UTC m=+51.158706093 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/45035743-85eb-4642-a5c9-adea5c5db4e1-metrics-tls") pod "dns-default-82dhq" (UID: "45035743-85eb-4642-a5c9-adea5c5db4e1") : secret "dns-default-metrics-tls" not found
Sep 17 13:56:17.088982 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:17.088947    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:56:17.225833 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:17.225805    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:56:18.094552 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:18.094518    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:56:18.199306 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:18.199273    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:56:19.914137 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:19.914105    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:56:20.570739 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:20.570706    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:56:20.724534 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:20.724499    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:56:20.882097 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:20.882011    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:56:21.473156 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:21.473123    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:56:21.584665 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:21.584638    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:56:22.268326 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:22.268288    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fb692cc1-e158-4666-baca-0d719d8c8aa6-registry-tls\") pod \"image-registry-d47ffd9b-tr5bs\" (UID: \"fb692cc1-e158-4666-baca-0d719d8c8aa6\") " pod="openshift-image-registry/image-registry-d47ffd9b-tr5bs"
Sep 17 13:56:22.268511 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:22.268353    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/45035743-85eb-4642-a5c9-adea5c5db4e1-metrics-tls\") pod \"dns-default-82dhq\" (UID: \"45035743-85eb-4642-a5c9-adea5c5db4e1\") " pod="openshift-dns/dns-default-82dhq"
Sep 17 13:56:22.268511 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:56:22.268434    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 13:56:22.268511 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:56:22.268440    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 13:56:22.268511 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:56:22.268474    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d47ffd9b-tr5bs: secret "image-registry-tls" not found
Sep 17 13:56:22.268511 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:56:22.268501    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45035743-85eb-4642-a5c9-adea5c5db4e1-metrics-tls podName:45035743-85eb-4642-a5c9-adea5c5db4e1 nodeName:}" failed. No retries permitted until 2026-09-17 13:56:38.268485046 +0000 UTC m=+67.210628693 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/45035743-85eb-4642-a5c9-adea5c5db4e1-metrics-tls") pod "dns-default-82dhq" (UID: "45035743-85eb-4642-a5c9-adea5c5db4e1") : secret "dns-default-metrics-tls" not found
Sep 17 13:56:22.268511 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:56:22.268515    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb692cc1-e158-4666-baca-0d719d8c8aa6-registry-tls podName:fb692cc1-e158-4666-baca-0d719d8c8aa6 nodeName:}" failed. No retries permitted until 2026-09-17 13:56:38.268509076 +0000 UTC m=+67.210652700 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fb692cc1-e158-4666-baca-0d719d8c8aa6-registry-tls") pod "image-registry-d47ffd9b-tr5bs" (UID: "fb692cc1-e158-4666-baca-0d719d8c8aa6") : secret "image-registry-tls" not found
Sep 17 13:56:24.390662 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:24.390620    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:56:32.027279 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:32.027243    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:56:32.127571 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:32.127539    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:56:32.226578 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:32.226549    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:56:32.807863 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:32.807836    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6546h"
Sep 17 13:56:33.031477 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:33.031430    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:56:35.076058 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:35.076014    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:56:35.567449 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:35.567417    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:56:35.678977 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:35.678948    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:56:37.369402 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:37.369365    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/14057e5c-c9fb-4894-869c-9780abc9c769-original-pull-secret\") pod \"global-pull-secret-syncer-58g8x\" (UID: \"14057e5c-c9fb-4894-869c-9780abc9c769\") " pod="kube-system/global-pull-secret-syncer-58g8x"
Sep 17 13:56:37.369402 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:37.369404    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d12b5b83-268b-404c-b77b-dd8ec7ea24d1-metrics-certs\") pod \"network-metrics-daemon-dqcvb\" (UID: \"d12b5b83-268b-404c-b77b-dd8ec7ea24d1\") " pod="openshift-multus/network-metrics-daemon-dqcvb"
Sep 17 13:56:37.369888 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:37.369435    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rndqv\" (UniqueName: \"kubernetes.io/projected/8677b05f-e355-4c69-b16e-8e64d8b0e701-kube-api-access-rndqv\") pod \"network-check-target-9jwwx\" (UID: \"8677b05f-e355-4c69-b16e-8e64d8b0e701\") " pod="openshift-network-diagnostics/network-check-target-9jwwx"
Sep 17 13:56:37.371775 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:37.371755    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 13:56:37.371775 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:37.371769    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 13:56:37.371845 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:37.371803    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 13:56:37.379670 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:56:37.379650    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 13:56:37.379776 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:56:37.379709    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d12b5b83-268b-404c-b77b-dd8ec7ea24d1-metrics-certs podName:d12b5b83-268b-404c-b77b-dd8ec7ea24d1 nodeName:}" failed. No retries permitted until 2026-09-17 13:57:41.379690139 +0000 UTC m=+130.321833762 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d12b5b83-268b-404c-b77b-dd8ec7ea24d1-metrics-certs") pod "network-metrics-daemon-dqcvb" (UID: "d12b5b83-268b-404c-b77b-dd8ec7ea24d1") : secret "metrics-daemon-secret" not found
Sep 17 13:56:37.381725 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:37.381708    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 13:56:37.382947 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:37.382925    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/14057e5c-c9fb-4894-869c-9780abc9c769-original-pull-secret\") pod \"global-pull-secret-syncer-58g8x\" (UID: \"14057e5c-c9fb-4894-869c-9780abc9c769\") " pod="kube-system/global-pull-secret-syncer-58g8x"
Sep 17 13:56:37.392411 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:37.392388    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rndqv\" (UniqueName: \"kubernetes.io/projected/8677b05f-e355-4c69-b16e-8e64d8b0e701-kube-api-access-rndqv\") pod \"network-check-target-9jwwx\" (UID: \"8677b05f-e355-4c69-b16e-8e64d8b0e701\") " pod="openshift-network-diagnostics/network-check-target-9jwwx"
Sep 17 13:56:37.474898 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:37.474868    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-cpdmf\""
Sep 17 13:56:37.479687 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:37.479671    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58g8x"
Sep 17 13:56:37.483541 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:37.483522    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9jwwx"
Sep 17 13:56:37.544062 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:37.544022    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:56:37.620964 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:37.620888    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:56:37.758272 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:37.758242    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-58g8x"]
Sep 17 13:56:37.760544 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:37.760521    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9jwwx"]
Sep 17 13:56:37.761659 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:56:37.761634    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14057e5c_c9fb_4894_869c_9780abc9c769.slice/crio-dede7f3f1207c0da1db0faf17d9517f80c9eb3b605f5b587afb4354b6f746a45 WatchSource:0}: Error finding container dede7f3f1207c0da1db0faf17d9517f80c9eb3b605f5b587afb4354b6f746a45: Status 404 returned error can't find the container with id dede7f3f1207c0da1db0faf17d9517f80c9eb3b605f5b587afb4354b6f746a45
Sep 17 13:56:37.762709 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:56:37.762688    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8677b05f_e355_4c69_b16e_8e64d8b0e701.slice/crio-61b60812471cf5d39c4d7fe0609f91a1a69e25beb3d2aaef3e7d554033079075 WatchSource:0}: Error finding container 61b60812471cf5d39c4d7fe0609f91a1a69e25beb3d2aaef3e7d554033079075: Status 404 returned error can't find the container with id 61b60812471cf5d39c4d7fe0609f91a1a69e25beb3d2aaef3e7d554033079075
Sep 17 13:56:37.893472 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:37.893371    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9jwwx" event={"ID":"8677b05f-e355-4c69-b16e-8e64d8b0e701","Type":"ContainerStarted","Data":"61b60812471cf5d39c4d7fe0609f91a1a69e25beb3d2aaef3e7d554033079075"}
Sep 17 13:56:37.894472 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:37.894428    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-58g8x" event={"ID":"14057e5c-c9fb-4894-869c-9780abc9c769","Type":"ContainerStarted","Data":"dede7f3f1207c0da1db0faf17d9517f80c9eb3b605f5b587afb4354b6f746a45"}
Sep 17 13:56:38.276714 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:38.276626    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/45035743-85eb-4642-a5c9-adea5c5db4e1-metrics-tls\") pod \"dns-default-82dhq\" (UID: \"45035743-85eb-4642-a5c9-adea5c5db4e1\") " pod="openshift-dns/dns-default-82dhq"
Sep 17 13:56:38.276714 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:38.276678    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fb692cc1-e158-4666-baca-0d719d8c8aa6-registry-tls\") pod \"image-registry-d47ffd9b-tr5bs\" (UID: \"fb692cc1-e158-4666-baca-0d719d8c8aa6\") " pod="openshift-image-registry/image-registry-d47ffd9b-tr5bs"
Sep 17 13:56:38.276887 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:56:38.276767    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 13:56:38.276887 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:56:38.276778    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d47ffd9b-tr5bs: secret "image-registry-tls" not found
Sep 17 13:56:38.276887 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:56:38.276779    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 13:56:38.276887 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:56:38.276825    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb692cc1-e158-4666-baca-0d719d8c8aa6-registry-tls podName:fb692cc1-e158-4666-baca-0d719d8c8aa6 nodeName:}" failed. No retries permitted until 2026-09-17 13:57:10.276812643 +0000 UTC m=+99.218956267 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fb692cc1-e158-4666-baca-0d719d8c8aa6-registry-tls") pod "image-registry-d47ffd9b-tr5bs" (UID: "fb692cc1-e158-4666-baca-0d719d8c8aa6") : secret "image-registry-tls" not found
Sep 17 13:56:38.276887 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:56:38.276836    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45035743-85eb-4642-a5c9-adea5c5db4e1-metrics-tls podName:45035743-85eb-4642-a5c9-adea5c5db4e1 nodeName:}" failed. No retries permitted until 2026-09-17 13:57:10.276831084 +0000 UTC m=+99.218974708 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/45035743-85eb-4642-a5c9-adea5c5db4e1-metrics-tls") pod "dns-default-82dhq" (UID: "45035743-85eb-4642-a5c9-adea5c5db4e1") : secret "dns-default-metrics-tls" not found
Sep 17 13:56:41.908066 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:41.908019    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9jwwx" event={"ID":"8677b05f-e355-4c69-b16e-8e64d8b0e701","Type":"ContainerStarted","Data":"5c4b80d195a08f3d096d6b74b29bc2a16037d2fa0bd9e090a9f0d2c31add3bab"}
Sep 17 13:56:41.908448 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:41.908125    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9jwwx"
Sep 17 13:56:41.928093 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:41.928024    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-9jwwx" podStartSLOduration=66.703132007 podStartE2EDuration="1m9.92800411s" totalImagesPullingTime="3.224872103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 13:55:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 13:56:41.92662457 +0000 UTC m=+70.868768216" watchObservedRunningTime="2026-09-17 13:56:41.92800411 +0000 UTC m=+70.870147757"
Sep 17 13:56:42.911929 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:42.911893    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-58g8x" event={"ID":"14057e5c-c9fb-4894-869c-9780abc9c769","Type":"ContainerStarted","Data":"cb87fd868b9c5823f5698b202cbb06d616ed3c6198f59df76d78e9464519820e"}
Sep 17 13:56:42.929537 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:42.929489    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-58g8x" podStartSLOduration=65.165345219 podStartE2EDuration="1m9.929474903s" totalImagesPullingTime="4.764129684s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 13:55:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 13:56:42.928209526 +0000 UTC m=+71.870353172" watchObservedRunningTime="2026-09-17 13:56:42.929474903 +0000 UTC m=+71.871618541"
Sep 17 13:56:55.031843 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:55.031716    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:56:55.095439 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:55.095395    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:56:55.174368 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:55.174342    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:56:56.577401 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:56.577357    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:56:56.659032 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:56.659005    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:56:59.575932 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:59.575893    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:56:59.660205 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:56:59.660167    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:10.293595 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:10.293563    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/45035743-85eb-4642-a5c9-adea5c5db4e1-metrics-tls\") pod \"dns-default-82dhq\" (UID: \"45035743-85eb-4642-a5c9-adea5c5db4e1\") " pod="openshift-dns/dns-default-82dhq"
Sep 17 13:57:10.294038 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:10.293605    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fb692cc1-e158-4666-baca-0d719d8c8aa6-registry-tls\") pod \"image-registry-d47ffd9b-tr5bs\" (UID: \"fb692cc1-e158-4666-baca-0d719d8c8aa6\") " pod="openshift-image-registry/image-registry-d47ffd9b-tr5bs"
Sep 17 13:57:10.294038 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:57:10.293715    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 13:57:10.294038 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:57:10.293753    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 13:57:10.294038 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:57:10.293763    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d47ffd9b-tr5bs: secret "image-registry-tls" not found
Sep 17 13:57:10.294038 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:57:10.293789    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45035743-85eb-4642-a5c9-adea5c5db4e1-metrics-tls podName:45035743-85eb-4642-a5c9-adea5c5db4e1 nodeName:}" failed. No retries permitted until 2026-09-17 13:58:14.293771137 +0000 UTC m=+163.235914764 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/45035743-85eb-4642-a5c9-adea5c5db4e1-metrics-tls") pod "dns-default-82dhq" (UID: "45035743-85eb-4642-a5c9-adea5c5db4e1") : secret "dns-default-metrics-tls" not found
Sep 17 13:57:10.294038 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:57:10.293808    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb692cc1-e158-4666-baca-0d719d8c8aa6-registry-tls podName:fb692cc1-e158-4666-baca-0d719d8c8aa6 nodeName:}" failed. No retries permitted until 2026-09-17 13:58:14.293797148 +0000 UTC m=+163.235940785 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fb692cc1-e158-4666-baca-0d719d8c8aa6-registry-tls") pod "image-registry-d47ffd9b-tr5bs" (UID: "fb692cc1-e158-4666-baca-0d719d8c8aa6") : secret "image-registry-tls" not found
Sep 17 13:57:12.360121 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:12.360090    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-997fdcdb-82rfx"]
Sep 17 13:57:12.363270 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:12.363253    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-997fdcdb-82rfx"
Sep 17 13:57:12.369250 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:12.369216    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 17 13:57:12.369250 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:12.369241    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 17 13:57:12.376904 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:12.376880    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 17 13:57:12.377009 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:12.376913    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 17 13:57:12.377009 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:12.376956    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 17 13:57:12.377606 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:12.377590    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 17 13:57:12.379047 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:12.379033    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-wjj9x\""
Sep 17 13:57:12.405997 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:12.405973    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/644fc778-b087-473e-91bf-44f9a89c2fef-stats-auth\") pod \"router-default-997fdcdb-82rfx\" (UID: \"644fc778-b087-473e-91bf-44f9a89c2fef\") " pod="openshift-ingress/router-default-997fdcdb-82rfx"
Sep 17 13:57:12.406083 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:12.406026    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/644fc778-b087-473e-91bf-44f9a89c2fef-default-certificate\") pod \"router-default-997fdcdb-82rfx\" (UID: \"644fc778-b087-473e-91bf-44f9a89c2fef\") " pod="openshift-ingress/router-default-997fdcdb-82rfx"
Sep 17 13:57:12.406132 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:12.406107    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/644fc778-b087-473e-91bf-44f9a89c2fef-service-ca-bundle\") pod \"router-default-997fdcdb-82rfx\" (UID: \"644fc778-b087-473e-91bf-44f9a89c2fef\") " pod="openshift-ingress/router-default-997fdcdb-82rfx"
Sep 17 13:57:12.406186 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:12.406163    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn2xr\" (UniqueName: \"kubernetes.io/projected/644fc778-b087-473e-91bf-44f9a89c2fef-kube-api-access-rn2xr\") pod \"router-default-997fdcdb-82rfx\" (UID: \"644fc778-b087-473e-91bf-44f9a89c2fef\") " pod="openshift-ingress/router-default-997fdcdb-82rfx"
Sep 17 13:57:12.406238 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:12.406212    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/644fc778-b087-473e-91bf-44f9a89c2fef-metrics-certs\") pod \"router-default-997fdcdb-82rfx\" (UID: \"644fc778-b087-473e-91bf-44f9a89c2fef\") " pod="openshift-ingress/router-default-997fdcdb-82rfx"
Sep 17 13:57:12.406798 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:12.406782    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-997fdcdb-82rfx"]
Sep 17 13:57:12.506472 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:12.506437    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/644fc778-b087-473e-91bf-44f9a89c2fef-stats-auth\") pod \"router-default-997fdcdb-82rfx\" (UID: \"644fc778-b087-473e-91bf-44f9a89c2fef\") " pod="openshift-ingress/router-default-997fdcdb-82rfx"
Sep 17 13:57:12.506561 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:12.506489    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/644fc778-b087-473e-91bf-44f9a89c2fef-default-certificate\") pod \"router-default-997fdcdb-82rfx\" (UID: \"644fc778-b087-473e-91bf-44f9a89c2fef\") " pod="openshift-ingress/router-default-997fdcdb-82rfx"
Sep 17 13:57:12.506609 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:12.506595    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/644fc778-b087-473e-91bf-44f9a89c2fef-service-ca-bundle\") pod \"router-default-997fdcdb-82rfx\" (UID: \"644fc778-b087-473e-91bf-44f9a89c2fef\") " pod="openshift-ingress/router-default-997fdcdb-82rfx"
Sep 17 13:57:12.506647 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:12.506625    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn2xr\" (UniqueName: \"kubernetes.io/projected/644fc778-b087-473e-91bf-44f9a89c2fef-kube-api-access-rn2xr\") pod \"router-default-997fdcdb-82rfx\" (UID: \"644fc778-b087-473e-91bf-44f9a89c2fef\") " pod="openshift-ingress/router-default-997fdcdb-82rfx"
Sep 17 13:57:12.506702 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:12.506656    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/644fc778-b087-473e-91bf-44f9a89c2fef-metrics-certs\") pod \"router-default-997fdcdb-82rfx\" (UID: \"644fc778-b087-473e-91bf-44f9a89c2fef\") " pod="openshift-ingress/router-default-997fdcdb-82rfx"
Sep 17 13:57:12.506770 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:57:12.506751    2560 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 13:57:12.506888 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:57:12.506789    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/644fc778-b087-473e-91bf-44f9a89c2fef-service-ca-bundle podName:644fc778-b087-473e-91bf-44f9a89c2fef nodeName:}" failed. No retries permitted until 2026-09-17 13:57:13.006772905 +0000 UTC m=+101.948916543 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/644fc778-b087-473e-91bf-44f9a89c2fef-service-ca-bundle") pod "router-default-997fdcdb-82rfx" (UID: "644fc778-b087-473e-91bf-44f9a89c2fef") : configmap references non-existent config key: service-ca.crt
Sep 17 13:57:12.506888 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:57:12.506813    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/644fc778-b087-473e-91bf-44f9a89c2fef-metrics-certs podName:644fc778-b087-473e-91bf-44f9a89c2fef nodeName:}" failed. No retries permitted until 2026-09-17 13:57:13.0068055 +0000 UTC m=+101.948949124 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/644fc778-b087-473e-91bf-44f9a89c2fef-metrics-certs") pod "router-default-997fdcdb-82rfx" (UID: "644fc778-b087-473e-91bf-44f9a89c2fef") : secret "router-metrics-certs-default" not found
Sep 17 13:57:12.508729 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:12.508710    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/644fc778-b087-473e-91bf-44f9a89c2fef-stats-auth\") pod \"router-default-997fdcdb-82rfx\" (UID: \"644fc778-b087-473e-91bf-44f9a89c2fef\") " pod="openshift-ingress/router-default-997fdcdb-82rfx"
Sep 17 13:57:12.508836 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:12.508743    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/644fc778-b087-473e-91bf-44f9a89c2fef-default-certificate\") pod \"router-default-997fdcdb-82rfx\" (UID: \"644fc778-b087-473e-91bf-44f9a89c2fef\") " pod="openshift-ingress/router-default-997fdcdb-82rfx"
Sep 17 13:57:12.521744 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:12.521724    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn2xr\" (UniqueName: \"kubernetes.io/projected/644fc778-b087-473e-91bf-44f9a89c2fef-kube-api-access-rn2xr\") pod \"router-default-997fdcdb-82rfx\" (UID: \"644fc778-b087-473e-91bf-44f9a89c2fef\") " pod="openshift-ingress/router-default-997fdcdb-82rfx"
Sep 17 13:57:12.652704 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:12.652645    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:12.728688 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:12.728668    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:12.914551 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:12.914498    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-9jwwx"
Sep 17 13:57:13.011592 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:13.011565    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/644fc778-b087-473e-91bf-44f9a89c2fef-service-ca-bundle\") pod \"router-default-997fdcdb-82rfx\" (UID: \"644fc778-b087-473e-91bf-44f9a89c2fef\") " pod="openshift-ingress/router-default-997fdcdb-82rfx"
Sep 17 13:57:13.011724 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:57:13.011681    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/644fc778-b087-473e-91bf-44f9a89c2fef-service-ca-bundle podName:644fc778-b087-473e-91bf-44f9a89c2fef nodeName:}" failed. No retries permitted until 2026-09-17 13:57:14.011668844 +0000 UTC m=+102.953812468 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/644fc778-b087-473e-91bf-44f9a89c2fef-service-ca-bundle") pod "router-default-997fdcdb-82rfx" (UID: "644fc778-b087-473e-91bf-44f9a89c2fef") : configmap references non-existent config key: service-ca.crt
Sep 17 13:57:13.011794 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:13.011759    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/644fc778-b087-473e-91bf-44f9a89c2fef-metrics-certs\") pod \"router-default-997fdcdb-82rfx\" (UID: \"644fc778-b087-473e-91bf-44f9a89c2fef\") " pod="openshift-ingress/router-default-997fdcdb-82rfx"
Sep 17 13:57:13.011889 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:57:13.011873    2560 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 13:57:13.011966 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:57:13.011955    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/644fc778-b087-473e-91bf-44f9a89c2fef-metrics-certs podName:644fc778-b087-473e-91bf-44f9a89c2fef nodeName:}" failed. No retries permitted until 2026-09-17 13:57:14.011938122 +0000 UTC m=+102.954081763 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/644fc778-b087-473e-91bf-44f9a89c2fef-metrics-certs") pod "router-default-997fdcdb-82rfx" (UID: "644fc778-b087-473e-91bf-44f9a89c2fef") : secret "router-metrics-certs-default" not found
Sep 17 13:57:14.016851 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:14.016810    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/644fc778-b087-473e-91bf-44f9a89c2fef-metrics-certs\") pod \"router-default-997fdcdb-82rfx\" (UID: \"644fc778-b087-473e-91bf-44f9a89c2fef\") " pod="openshift-ingress/router-default-997fdcdb-82rfx"
Sep 17 13:57:14.017210 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:14.016897    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/644fc778-b087-473e-91bf-44f9a89c2fef-service-ca-bundle\") pod \"router-default-997fdcdb-82rfx\" (UID: \"644fc778-b087-473e-91bf-44f9a89c2fef\") " pod="openshift-ingress/router-default-997fdcdb-82rfx"
Sep 17 13:57:14.017210 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:57:14.016961    2560 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 13:57:14.017210 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:57:14.017007    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/644fc778-b087-473e-91bf-44f9a89c2fef-service-ca-bundle podName:644fc778-b087-473e-91bf-44f9a89c2fef nodeName:}" failed. No retries permitted until 2026-09-17 13:57:16.0169937 +0000 UTC m=+104.959137345 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/644fc778-b087-473e-91bf-44f9a89c2fef-service-ca-bundle") pod "router-default-997fdcdb-82rfx" (UID: "644fc778-b087-473e-91bf-44f9a89c2fef") : configmap references non-existent config key: service-ca.crt
Sep 17 13:57:14.017210 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:57:14.017023    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/644fc778-b087-473e-91bf-44f9a89c2fef-metrics-certs podName:644fc778-b087-473e-91bf-44f9a89c2fef nodeName:}" failed. No retries permitted until 2026-09-17 13:57:16.017017706 +0000 UTC m=+104.959161334 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/644fc778-b087-473e-91bf-44f9a89c2fef-metrics-certs") pod "router-default-997fdcdb-82rfx" (UID: "644fc778-b087-473e-91bf-44f9a89c2fef") : secret "router-metrics-certs-default" not found
Sep 17 13:57:15.942909 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:15.942879    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-4848q"]
Sep 17 13:57:15.945397 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:15.945380    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4848q"
Sep 17 13:57:15.949698 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:15.949675    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 13:57:15.950110 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:15.950090    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 13:57:15.950110 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:15.950099    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 13:57:15.950236 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:15.950089    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 13:57:15.950236 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:15.950132    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-twkdq\""
Sep 17 13:57:15.961820 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:15.961801    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4848q"]
Sep 17 13:57:16.030745 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:16.030718    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3e8dc684-ddfc-4801-a7de-7874e92eb31f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4848q\" (UID: \"3e8dc684-ddfc-4801-a7de-7874e92eb31f\") " pod="openshift-insights/insights-runtime-extractor-4848q"
Sep 17 13:57:16.031141 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:16.030751    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtc5x\" (UniqueName: \"kubernetes.io/projected/3e8dc684-ddfc-4801-a7de-7874e92eb31f-kube-api-access-gtc5x\") pod \"insights-runtime-extractor-4848q\" (UID: \"3e8dc684-ddfc-4801-a7de-7874e92eb31f\") " pod="openshift-insights/insights-runtime-extractor-4848q"
Sep 17 13:57:16.031231 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:16.031164    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/644fc778-b087-473e-91bf-44f9a89c2fef-service-ca-bundle\") pod \"router-default-997fdcdb-82rfx\" (UID: \"644fc778-b087-473e-91bf-44f9a89c2fef\") " pod="openshift-ingress/router-default-997fdcdb-82rfx"
Sep 17 13:57:16.031231 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:16.031212    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3e8dc684-ddfc-4801-a7de-7874e92eb31f-data-volume\") pod \"insights-runtime-extractor-4848q\" (UID: \"3e8dc684-ddfc-4801-a7de-7874e92eb31f\") " pod="openshift-insights/insights-runtime-extractor-4848q"
Sep 17 13:57:16.031343 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:16.031282    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3e8dc684-ddfc-4801-a7de-7874e92eb31f-crio-socket\") pod \"insights-runtime-extractor-4848q\" (UID: \"3e8dc684-ddfc-4801-a7de-7874e92eb31f\") " pod="openshift-insights/insights-runtime-extractor-4848q"
Sep 17 13:57:16.031397 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:16.031367    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/644fc778-b087-473e-91bf-44f9a89c2fef-metrics-certs\") pod \"router-default-997fdcdb-82rfx\" (UID: \"644fc778-b087-473e-91bf-44f9a89c2fef\") " pod="openshift-ingress/router-default-997fdcdb-82rfx"
Sep 17 13:57:16.031397 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:57:16.031378    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/644fc778-b087-473e-91bf-44f9a89c2fef-service-ca-bundle podName:644fc778-b087-473e-91bf-44f9a89c2fef nodeName:}" failed. No retries permitted until 2026-09-17 13:57:20.031358572 +0000 UTC m=+108.973502214 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/644fc778-b087-473e-91bf-44f9a89c2fef-service-ca-bundle") pod "router-default-997fdcdb-82rfx" (UID: "644fc778-b087-473e-91bf-44f9a89c2fef") : configmap references non-existent config key: service-ca.crt
Sep 17 13:57:16.031524 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:16.031417    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3e8dc684-ddfc-4801-a7de-7874e92eb31f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4848q\" (UID: \"3e8dc684-ddfc-4801-a7de-7874e92eb31f\") " pod="openshift-insights/insights-runtime-extractor-4848q"
Sep 17 13:57:16.031524 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:57:16.031483    2560 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 13:57:16.034174 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:57:16.031778    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/644fc778-b087-473e-91bf-44f9a89c2fef-metrics-certs podName:644fc778-b087-473e-91bf-44f9a89c2fef nodeName:}" failed. No retries permitted until 2026-09-17 13:57:20.031754262 +0000 UTC m=+108.973897927 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/644fc778-b087-473e-91bf-44f9a89c2fef-metrics-certs") pod "router-default-997fdcdb-82rfx" (UID: "644fc778-b087-473e-91bf-44f9a89c2fef") : secret "router-metrics-certs-default" not found
Sep 17 13:57:16.099853 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:16.099830    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:16.131666 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:16.131646    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3e8dc684-ddfc-4801-a7de-7874e92eb31f-crio-socket\") pod \"insights-runtime-extractor-4848q\" (UID: \"3e8dc684-ddfc-4801-a7de-7874e92eb31f\") " pod="openshift-insights/insights-runtime-extractor-4848q"
Sep 17 13:57:16.131737 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:16.131691    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3e8dc684-ddfc-4801-a7de-7874e92eb31f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4848q\" (UID: \"3e8dc684-ddfc-4801-a7de-7874e92eb31f\") " pod="openshift-insights/insights-runtime-extractor-4848q"
Sep 17 13:57:16.131737 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:16.131730    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3e8dc684-ddfc-4801-a7de-7874e92eb31f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4848q\" (UID: \"3e8dc684-ddfc-4801-a7de-7874e92eb31f\") " pod="openshift-insights/insights-runtime-extractor-4848q"
Sep 17 13:57:16.131811 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:16.131749    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtc5x\" (UniqueName: \"kubernetes.io/projected/3e8dc684-ddfc-4801-a7de-7874e92eb31f-kube-api-access-gtc5x\") pod \"insights-runtime-extractor-4848q\" (UID: \"3e8dc684-ddfc-4801-a7de-7874e92eb31f\") " pod="openshift-insights/insights-runtime-extractor-4848q"
Sep 17 13:57:16.131811 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:16.131756    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3e8dc684-ddfc-4801-a7de-7874e92eb31f-crio-socket\") pod \"insights-runtime-extractor-4848q\" (UID: \"3e8dc684-ddfc-4801-a7de-7874e92eb31f\") " pod="openshift-insights/insights-runtime-extractor-4848q"
Sep 17 13:57:16.131811 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:16.131774    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3e8dc684-ddfc-4801-a7de-7874e92eb31f-data-volume\") pod \"insights-runtime-extractor-4848q\" (UID: \"3e8dc684-ddfc-4801-a7de-7874e92eb31f\") " pod="openshift-insights/insights-runtime-extractor-4848q"
Sep 17 13:57:16.131914 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:57:16.131858    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 13:57:16.131914 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:57:16.131902    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 13:57:16.131986 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:57:16.131916    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3e8dc684-ddfc-4801-a7de-7874e92eb31f-kube-rbac-proxy-cm podName:3e8dc684-ddfc-4801-a7de-7874e92eb31f nodeName:}" failed. No retries permitted until 2026-09-17 13:57:16.631900625 +0000 UTC m=+105.574044265 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3e8dc684-ddfc-4801-a7de-7874e92eb31f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4848q" (UID: "3e8dc684-ddfc-4801-a7de-7874e92eb31f") : configmap "kube-rbac-proxy" not found
Sep 17 13:57:16.131986 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:57:16.131949    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e8dc684-ddfc-4801-a7de-7874e92eb31f-insights-runtime-extractor-tls podName:3e8dc684-ddfc-4801-a7de-7874e92eb31f nodeName:}" failed. No retries permitted until 2026-09-17 13:57:16.631937739 +0000 UTC m=+105.574081362 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3e8dc684-ddfc-4801-a7de-7874e92eb31f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4848q" (UID: "3e8dc684-ddfc-4801-a7de-7874e92eb31f") : secret "insights-runtime-extractor-tls" not found
Sep 17 13:57:16.132070 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:16.132027    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3e8dc684-ddfc-4801-a7de-7874e92eb31f-data-volume\") pod \"insights-runtime-extractor-4848q\" (UID: \"3e8dc684-ddfc-4801-a7de-7874e92eb31f\") " pod="openshift-insights/insights-runtime-extractor-4848q"
Sep 17 13:57:16.141750 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:16.141734    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtc5x\" (UniqueName: \"kubernetes.io/projected/3e8dc684-ddfc-4801-a7de-7874e92eb31f-kube-api-access-gtc5x\") pod \"insights-runtime-extractor-4848q\" (UID: \"3e8dc684-ddfc-4801-a7de-7874e92eb31f\") " pod="openshift-insights/insights-runtime-extractor-4848q"
Sep 17 13:57:16.224438 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:16.224379    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:16.635505 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:16.635473    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3e8dc684-ddfc-4801-a7de-7874e92eb31f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4848q\" (UID: \"3e8dc684-ddfc-4801-a7de-7874e92eb31f\") " pod="openshift-insights/insights-runtime-extractor-4848q"
Sep 17 13:57:16.635654 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:16.635528    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3e8dc684-ddfc-4801-a7de-7874e92eb31f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4848q\" (UID: \"3e8dc684-ddfc-4801-a7de-7874e92eb31f\") " pod="openshift-insights/insights-runtime-extractor-4848q"
Sep 17 13:57:16.635654 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:57:16.635596    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 13:57:16.635654 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:57:16.635623    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 13:57:16.635757 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:57:16.635672    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3e8dc684-ddfc-4801-a7de-7874e92eb31f-kube-rbac-proxy-cm podName:3e8dc684-ddfc-4801-a7de-7874e92eb31f nodeName:}" failed. No retries permitted until 2026-09-17 13:57:17.63565622 +0000 UTC m=+106.577799864 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3e8dc684-ddfc-4801-a7de-7874e92eb31f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4848q" (UID: "3e8dc684-ddfc-4801-a7de-7874e92eb31f") : configmap "kube-rbac-proxy" not found
Sep 17 13:57:16.635757 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:57:16.635689    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e8dc684-ddfc-4801-a7de-7874e92eb31f-insights-runtime-extractor-tls podName:3e8dc684-ddfc-4801-a7de-7874e92eb31f nodeName:}" failed. No retries permitted until 2026-09-17 13:57:17.635681158 +0000 UTC m=+106.577824782 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3e8dc684-ddfc-4801-a7de-7874e92eb31f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4848q" (UID: "3e8dc684-ddfc-4801-a7de-7874e92eb31f") : secret "insights-runtime-extractor-tls" not found
Sep 17 13:57:17.641918 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:17.641887    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3e8dc684-ddfc-4801-a7de-7874e92eb31f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4848q\" (UID: \"3e8dc684-ddfc-4801-a7de-7874e92eb31f\") " pod="openshift-insights/insights-runtime-extractor-4848q"
Sep 17 13:57:17.642262 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:17.641940    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3e8dc684-ddfc-4801-a7de-7874e92eb31f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4848q\" (UID: \"3e8dc684-ddfc-4801-a7de-7874e92eb31f\") " pod="openshift-insights/insights-runtime-extractor-4848q"
Sep 17 13:57:17.642262 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:57:17.641996    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 13:57:17.642262 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:57:17.642034    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 13:57:17.642262 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:57:17.642044    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3e8dc684-ddfc-4801-a7de-7874e92eb31f-kube-rbac-proxy-cm podName:3e8dc684-ddfc-4801-a7de-7874e92eb31f nodeName:}" failed. No retries permitted until 2026-09-17 13:57:19.642031644 +0000 UTC m=+108.584175290 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3e8dc684-ddfc-4801-a7de-7874e92eb31f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4848q" (UID: "3e8dc684-ddfc-4801-a7de-7874e92eb31f") : configmap "kube-rbac-proxy" not found
Sep 17 13:57:17.642262 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:57:17.642073    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e8dc684-ddfc-4801-a7de-7874e92eb31f-insights-runtime-extractor-tls podName:3e8dc684-ddfc-4801-a7de-7874e92eb31f nodeName:}" failed. No retries permitted until 2026-09-17 13:57:19.642062978 +0000 UTC m=+108.584206602 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3e8dc684-ddfc-4801-a7de-7874e92eb31f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4848q" (UID: "3e8dc684-ddfc-4801-a7de-7874e92eb31f") : secret "insights-runtime-extractor-tls" not found
Sep 17 13:57:17.669812 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:17.669790    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2zqvg_a4a8010a-d12b-4b1a-a0db-e1c64109b151/dns-node-resolver/0.log"
Sep 17 13:57:18.467957 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:18.467927    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9wpxw_9fcb9893-714e-47b5-9b6f-37e706357ffe/node-ca/0.log"
Sep 17 13:57:19.465919 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:19.465890    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wwqnd_2769cbbe-3a80-48c0-bf6b-a993360d1ca7/serve-healthcheck-canary/0.log"
Sep 17 13:57:19.578428 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:19.578394    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:19.655409 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:19.655385    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3e8dc684-ddfc-4801-a7de-7874e92eb31f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4848q\" (UID: \"3e8dc684-ddfc-4801-a7de-7874e92eb31f\") " pod="openshift-insights/insights-runtime-extractor-4848q"
Sep 17 13:57:19.655537 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:19.655448    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3e8dc684-ddfc-4801-a7de-7874e92eb31f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4848q\" (UID: \"3e8dc684-ddfc-4801-a7de-7874e92eb31f\") " pod="openshift-insights/insights-runtime-extractor-4848q"
Sep 17 13:57:19.655537 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:57:19.655524    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 13:57:19.655609 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:57:19.655536    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 13:57:19.655609 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:57:19.655571    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3e8dc684-ddfc-4801-a7de-7874e92eb31f-kube-rbac-proxy-cm podName:3e8dc684-ddfc-4801-a7de-7874e92eb31f nodeName:}" failed. No retries permitted until 2026-09-17 13:57:23.655559877 +0000 UTC m=+112.597703501 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3e8dc684-ddfc-4801-a7de-7874e92eb31f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4848q" (UID: "3e8dc684-ddfc-4801-a7de-7874e92eb31f") : configmap "kube-rbac-proxy" not found
Sep 17 13:57:19.655609 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:57:19.655586    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e8dc684-ddfc-4801-a7de-7874e92eb31f-insights-runtime-extractor-tls podName:3e8dc684-ddfc-4801-a7de-7874e92eb31f nodeName:}" failed. No retries permitted until 2026-09-17 13:57:23.655577818 +0000 UTC m=+112.597721442 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3e8dc684-ddfc-4801-a7de-7874e92eb31f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4848q" (UID: "3e8dc684-ddfc-4801-a7de-7874e92eb31f") : secret "insights-runtime-extractor-tls" not found
Sep 17 13:57:19.682095 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:19.682071    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:20.058238 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:20.058207    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/644fc778-b087-473e-91bf-44f9a89c2fef-service-ca-bundle\") pod \"router-default-997fdcdb-82rfx\" (UID: \"644fc778-b087-473e-91bf-44f9a89c2fef\") " pod="openshift-ingress/router-default-997fdcdb-82rfx"
Sep 17 13:57:20.058409 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:20.058268    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/644fc778-b087-473e-91bf-44f9a89c2fef-metrics-certs\") pod \"router-default-997fdcdb-82rfx\" (UID: \"644fc778-b087-473e-91bf-44f9a89c2fef\") " pod="openshift-ingress/router-default-997fdcdb-82rfx"
Sep 17 13:57:20.058409 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:57:20.058352    2560 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 13:57:20.058409 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:57:20.058362    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/644fc778-b087-473e-91bf-44f9a89c2fef-service-ca-bundle podName:644fc778-b087-473e-91bf-44f9a89c2fef nodeName:}" failed. No retries permitted until 2026-09-17 13:57:28.058345983 +0000 UTC m=+117.000489608 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/644fc778-b087-473e-91bf-44f9a89c2fef-service-ca-bundle") pod "router-default-997fdcdb-82rfx" (UID: "644fc778-b087-473e-91bf-44f9a89c2fef") : configmap references non-existent config key: service-ca.crt
Sep 17 13:57:20.058409 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:57:20.058410    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/644fc778-b087-473e-91bf-44f9a89c2fef-metrics-certs podName:644fc778-b087-473e-91bf-44f9a89c2fef nodeName:}" failed. No retries permitted until 2026-09-17 13:57:28.058398161 +0000 UTC m=+117.000541785 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/644fc778-b087-473e-91bf-44f9a89c2fef-metrics-certs") pod "router-default-997fdcdb-82rfx" (UID: "644fc778-b087-473e-91bf-44f9a89c2fef") : secret "router-metrics-certs-default" not found
Sep 17 13:57:23.378987 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:23.378952    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-bshpr"]
Sep 17 13:57:23.381739 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:23.381722    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-bshpr"
Sep 17 13:57:23.384146 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:23.384119    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-kzjq7\""
Sep 17 13:57:23.392370 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:23.392351    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-bshpr"]
Sep 17 13:57:23.479632 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:23.479610    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc7qn\" (UniqueName: \"kubernetes.io/projected/4bc3aa88-46c1-4eb2-a29a-f4ebdbdca984-kube-api-access-jc7qn\") pod \"network-check-source-6f8b768455-bshpr\" (UID: \"4bc3aa88-46c1-4eb2-a29a-f4ebdbdca984\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-bshpr"
Sep 17 13:57:23.580610 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:23.580587    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc7qn\" (UniqueName: \"kubernetes.io/projected/4bc3aa88-46c1-4eb2-a29a-f4ebdbdca984-kube-api-access-jc7qn\") pod \"network-check-source-6f8b768455-bshpr\" (UID: \"4bc3aa88-46c1-4eb2-a29a-f4ebdbdca984\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-bshpr"
Sep 17 13:57:23.591925 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:23.591903    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc7qn\" (UniqueName: \"kubernetes.io/projected/4bc3aa88-46c1-4eb2-a29a-f4ebdbdca984-kube-api-access-jc7qn\") pod \"network-check-source-6f8b768455-bshpr\" (UID: \"4bc3aa88-46c1-4eb2-a29a-f4ebdbdca984\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-bshpr"
Sep 17 13:57:23.681092 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:23.681033    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3e8dc684-ddfc-4801-a7de-7874e92eb31f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4848q\" (UID: \"3e8dc684-ddfc-4801-a7de-7874e92eb31f\") " pod="openshift-insights/insights-runtime-extractor-4848q"
Sep 17 13:57:23.681092 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:23.681088    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3e8dc684-ddfc-4801-a7de-7874e92eb31f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4848q\" (UID: \"3e8dc684-ddfc-4801-a7de-7874e92eb31f\") " pod="openshift-insights/insights-runtime-extractor-4848q"
Sep 17 13:57:23.681248 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:57:23.681162    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 13:57:23.681248 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:57:23.681224    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3e8dc684-ddfc-4801-a7de-7874e92eb31f-kube-rbac-proxy-cm podName:3e8dc684-ddfc-4801-a7de-7874e92eb31f nodeName:}" failed. No retries permitted until 2026-09-17 13:57:31.681203144 +0000 UTC m=+120.623346784 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3e8dc684-ddfc-4801-a7de-7874e92eb31f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4848q" (UID: "3e8dc684-ddfc-4801-a7de-7874e92eb31f") : configmap "kube-rbac-proxy" not found
Sep 17 13:57:23.681248 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:57:23.681169    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 13:57:23.681372 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:57:23.681278    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e8dc684-ddfc-4801-a7de-7874e92eb31f-insights-runtime-extractor-tls podName:3e8dc684-ddfc-4801-a7de-7874e92eb31f nodeName:}" failed. No retries permitted until 2026-09-17 13:57:31.681266723 +0000 UTC m=+120.623410347 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3e8dc684-ddfc-4801-a7de-7874e92eb31f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4848q" (UID: "3e8dc684-ddfc-4801-a7de-7874e92eb31f") : secret "insights-runtime-extractor-tls" not found
Sep 17 13:57:23.690623 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:23.690539    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-bshpr"
Sep 17 13:57:23.961590 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:23.961517    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-bshpr"]
Sep 17 13:57:23.965381 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:57:23.965354    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bc3aa88_46c1_4eb2_a29a_f4ebdbdca984.slice/crio-cbd648adf6baec784fb466c47f257dd790af5d25198bc5e438b96a02827774a3 WatchSource:0}: Error finding container cbd648adf6baec784fb466c47f257dd790af5d25198bc5e438b96a02827774a3: Status 404 returned error can't find the container with id cbd648adf6baec784fb466c47f257dd790af5d25198bc5e438b96a02827774a3
Sep 17 13:57:24.002762 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:24.002742    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-bshpr" event={"ID":"4bc3aa88-46c1-4eb2-a29a-f4ebdbdca984","Type":"ContainerStarted","Data":"cbd648adf6baec784fb466c47f257dd790af5d25198bc5e438b96a02827774a3"}
Sep 17 13:57:25.006978 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:25.006941    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-bshpr" event={"ID":"4bc3aa88-46c1-4eb2-a29a-f4ebdbdca984","Type":"ContainerStarted","Data":"d3dafecc4090fd8d1a857dfde47d4e583121ada8c952e7a131bbad8337c19337"}
Sep 17 13:57:25.029408 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:25.029364    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-bshpr" podStartSLOduration=2.029350901 podStartE2EDuration="2.029350901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 13:57:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 13:57:25.028384557 +0000 UTC m=+113.970528203" watchObservedRunningTime="2026-09-17 13:57:25.029350901 +0000 UTC m=+113.971494547"
Sep 17 13:57:26.466590 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:26.466554    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:26.566623 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:26.566591    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:26.694553 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:26.694525    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:27.169478 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:27.169439    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:27.268776 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:27.268749    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:27.507165 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:27.507082    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:27.612320 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:27.612297    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:28.110930 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:28.110902    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/644fc778-b087-473e-91bf-44f9a89c2fef-service-ca-bundle\") pod \"router-default-997fdcdb-82rfx\" (UID: \"644fc778-b087-473e-91bf-44f9a89c2fef\") " pod="openshift-ingress/router-default-997fdcdb-82rfx"
Sep 17 13:57:28.111080 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:28.110952    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/644fc778-b087-473e-91bf-44f9a89c2fef-metrics-certs\") pod \"router-default-997fdcdb-82rfx\" (UID: \"644fc778-b087-473e-91bf-44f9a89c2fef\") " pod="openshift-ingress/router-default-997fdcdb-82rfx"
Sep 17 13:57:28.111080 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:57:28.111053    2560 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 13:57:28.111145 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:57:28.111102    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/644fc778-b087-473e-91bf-44f9a89c2fef-metrics-certs podName:644fc778-b087-473e-91bf-44f9a89c2fef nodeName:}" failed. No retries permitted until 2026-09-17 13:57:44.111089901 +0000 UTC m=+133.053233524 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/644fc778-b087-473e-91bf-44f9a89c2fef-metrics-certs") pod "router-default-997fdcdb-82rfx" (UID: "644fc778-b087-473e-91bf-44f9a89c2fef") : secret "router-metrics-certs-default" not found
Sep 17 13:57:28.111192 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:57:28.111181    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/644fc778-b087-473e-91bf-44f9a89c2fef-service-ca-bundle podName:644fc778-b087-473e-91bf-44f9a89c2fef nodeName:}" failed. No retries permitted until 2026-09-17 13:57:44.111166027 +0000 UTC m=+133.053309683 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/644fc778-b087-473e-91bf-44f9a89c2fef-service-ca-bundle") pod "router-default-997fdcdb-82rfx" (UID: "644fc778-b087-473e-91bf-44f9a89c2fef") : configmap references non-existent config key: service-ca.crt
Sep 17 13:57:28.873688 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:28.873656    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:28.972762 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:28.972730    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:29.189266 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:29.189191    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:30.044828 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:30.044790    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:31.348895 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:31.348859    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:31.666770 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:31.666689    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:31.735164 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:31.735140    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3e8dc684-ddfc-4801-a7de-7874e92eb31f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4848q\" (UID: \"3e8dc684-ddfc-4801-a7de-7874e92eb31f\") " pod="openshift-insights/insights-runtime-extractor-4848q"
Sep 17 13:57:31.735295 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:31.735198    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3e8dc684-ddfc-4801-a7de-7874e92eb31f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4848q\" (UID: \"3e8dc684-ddfc-4801-a7de-7874e92eb31f\") " pod="openshift-insights/insights-runtime-extractor-4848q"
Sep 17 13:57:31.735336 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:57:31.735325    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 13:57:31.735397 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:57:31.735386    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e8dc684-ddfc-4801-a7de-7874e92eb31f-insights-runtime-extractor-tls podName:3e8dc684-ddfc-4801-a7de-7874e92eb31f nodeName:}" failed. No retries permitted until 2026-09-17 13:57:47.735368584 +0000 UTC m=+136.677512221 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3e8dc684-ddfc-4801-a7de-7874e92eb31f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4848q" (UID: "3e8dc684-ddfc-4801-a7de-7874e92eb31f") : secret "insights-runtime-extractor-tls" not found
Sep 17 13:57:31.735725 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:31.735706    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3e8dc684-ddfc-4801-a7de-7874e92eb31f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4848q\" (UID: \"3e8dc684-ddfc-4801-a7de-7874e92eb31f\") " pod="openshift-insights/insights-runtime-extractor-4848q"
Sep 17 13:57:31.804430 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:31.804388    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:32.287396 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:32.287361    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:32.413378 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:32.413351    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:32.505639 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:32.505610    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:33.460182 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:33.460127    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:33.672660 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:33.672622    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:34.058384 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:34.058352    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:34.253612 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:34.253583    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:34.478820 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:34.478739    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:35.025834 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:35.025802    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:35.132056 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:35.132022    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:35.235031 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:35.234998    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:35.485259 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:35.485233    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:35.862915 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:35.862884    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:36.051061 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:36.051028    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:36.125802 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:36.125706    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:36.258838 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:36.258812    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:36.939418 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:36.939384    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:37.251251 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:37.251170    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:37.852679 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:37.852648    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:38.054201 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:38.054166    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:38.263925 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:38.263839    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:38.441455 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:38.441428    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:38.781818 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:38.781787    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:39.463545 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:39.463515    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:39.654736 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:39.654705    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:39.861996 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:39.861965    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:40.058980 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:40.058945    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:40.320377 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:40.320345    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:40.401997 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:40.401965    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:40.551920 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:40.551884    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:40.652500 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:40.652410    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:40.752918 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:40.752878    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:40.852381 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:40.852352    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:41.062128 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:41.062087    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:41.159150 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:41.159117    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:41.258434 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:41.258395    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:41.359450 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:41.359384    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:41.405603 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:41.405580    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d12b5b83-268b-404c-b77b-dd8ec7ea24d1-metrics-certs\") pod \"network-metrics-daemon-dqcvb\" (UID: \"d12b5b83-268b-404c-b77b-dd8ec7ea24d1\") " pod="openshift-multus/network-metrics-daemon-dqcvb"
Sep 17 13:57:41.407740 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:41.407717    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d12b5b83-268b-404c-b77b-dd8ec7ea24d1-metrics-certs\") pod \"network-metrics-daemon-dqcvb\" (UID: \"d12b5b83-268b-404c-b77b-dd8ec7ea24d1\") " pod="openshift-multus/network-metrics-daemon-dqcvb"
Sep 17 13:57:41.460343 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:41.460310    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:41.560011 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:41.559973    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:41.687049 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:41.686982    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gs47f\""
Sep 17 13:57:41.694947 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:41.694928    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqcvb"
Sep 17 13:57:41.772337 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:41.772294    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:41.872610 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:41.872569    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:41.966941 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:41.966883    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dqcvb"]
Sep 17 13:57:41.969734 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:57:41.969712    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd12b5b83_268b_404c_b77b_dd8ec7ea24d1.slice/crio-e37cda9beed2efd0546f2b825b759232e5ff550c1bba54ad032b2928702e157a WatchSource:0}: Error finding container e37cda9beed2efd0546f2b825b759232e5ff550c1bba54ad032b2928702e157a: Status 404 returned error can't find the container with id e37cda9beed2efd0546f2b825b759232e5ff550c1bba54ad032b2928702e157a
Sep 17 13:57:41.971007 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:41.970982    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:42.045680 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:42.045658    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dqcvb" event={"ID":"d12b5b83-268b-404c-b77b-dd8ec7ea24d1","Type":"ContainerStarted","Data":"e37cda9beed2efd0546f2b825b759232e5ff550c1bba54ad032b2928702e157a"}
Sep 17 13:57:42.071008 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:42.070982    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:42.172736 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:42.172707    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:42.272897 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:42.272827    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:42.448188 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:42.448155    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:43.050989 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:43.050958    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dqcvb" event={"ID":"d12b5b83-268b-404c-b77b-dd8ec7ea24d1","Type":"ContainerStarted","Data":"cb6c5f5e46cbe1b260bfe580cf14a19728ec81be1a1a1a3ae59e8cb5cd9f4172"}
Sep 17 13:57:43.416682 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:43.416647    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:44.056956 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:44.056923    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dqcvb" event={"ID":"d12b5b83-268b-404c-b77b-dd8ec7ea24d1","Type":"ContainerStarted","Data":"be0b2bf81d31d82893b714d9cbc307a09471ff9d6edc672e9fbdfaa2c0577ec1"}
Sep 17 13:57:44.086282 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:44.086239    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-dqcvb" podStartSLOduration=131.120483469 podStartE2EDuration="2m12.086225672s" totalImagesPullingTime="965.742203ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 13:55:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 13:57:44.083950937 +0000 UTC m=+133.026094583" watchObservedRunningTime="2026-09-17 13:57:44.086225672 +0000 UTC m=+133.028369317"
Sep 17 13:57:44.124141 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:44.124110    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/644fc778-b087-473e-91bf-44f9a89c2fef-service-ca-bundle\") pod \"router-default-997fdcdb-82rfx\" (UID: \"644fc778-b087-473e-91bf-44f9a89c2fef\") " pod="openshift-ingress/router-default-997fdcdb-82rfx"
Sep 17 13:57:44.124256 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:44.124152    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/644fc778-b087-473e-91bf-44f9a89c2fef-metrics-certs\") pod \"router-default-997fdcdb-82rfx\" (UID: \"644fc778-b087-473e-91bf-44f9a89c2fef\") " pod="openshift-ingress/router-default-997fdcdb-82rfx"
Sep 17 13:57:44.124838 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:44.124820    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/644fc778-b087-473e-91bf-44f9a89c2fef-service-ca-bundle\") pod \"router-default-997fdcdb-82rfx\" (UID: \"644fc778-b087-473e-91bf-44f9a89c2fef\") " pod="openshift-ingress/router-default-997fdcdb-82rfx"
Sep 17 13:57:44.126372 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:44.126357    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/644fc778-b087-473e-91bf-44f9a89c2fef-metrics-certs\") pod \"router-default-997fdcdb-82rfx\" (UID: \"644fc778-b087-473e-91bf-44f9a89c2fef\") " pod="openshift-ingress/router-default-997fdcdb-82rfx"
Sep 17 13:57:44.173896 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:44.173874    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-wjj9x\""
Sep 17 13:57:44.181858 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:44.181843    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-997fdcdb-82rfx"
Sep 17 13:57:44.462075 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:44.462002    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-997fdcdb-82rfx"]
Sep 17 13:57:44.465577 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:57:44.465544    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod644fc778_b087_473e_91bf_44f9a89c2fef.slice/crio-2d1624c47ee61d3effa3a6da8ff0d942e935e6837cc179836eb2c8fd2db6ed58 WatchSource:0}: Error finding container 2d1624c47ee61d3effa3a6da8ff0d942e935e6837cc179836eb2c8fd2db6ed58: Status 404 returned error can't find the container with id 2d1624c47ee61d3effa3a6da8ff0d942e935e6837cc179836eb2c8fd2db6ed58
Sep 17 13:57:45.062243 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:45.062203    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-997fdcdb-82rfx" event={"ID":"644fc778-b087-473e-91bf-44f9a89c2fef","Type":"ContainerStarted","Data":"2cc6a6319c390ce59f6aa0af236a83909c2928534a75aa6bc99a50b48da6ec7b"}
Sep 17 13:57:45.062688 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:45.062256    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-997fdcdb-82rfx" event={"ID":"644fc778-b087-473e-91bf-44f9a89c2fef","Type":"ContainerStarted","Data":"2d1624c47ee61d3effa3a6da8ff0d942e935e6837cc179836eb2c8fd2db6ed58"}
Sep 17 13:57:45.088917 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:45.088876    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-997fdcdb-82rfx" podStartSLOduration=33.088860606 podStartE2EDuration="33.088860606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 13:57:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 13:57:45.088006722 +0000 UTC m=+134.030150368" watchObservedRunningTime="2026-09-17 13:57:45.088860606 +0000 UTC m=+134.031004251"
Sep 17 13:57:45.182884 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:45.182855    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-997fdcdb-82rfx"
Sep 17 13:57:45.185328 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:45.185311    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-997fdcdb-82rfx"
Sep 17 13:57:46.065773 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:46.065744    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-997fdcdb-82rfx"
Sep 17 13:57:46.066966 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:46.066943    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-997fdcdb-82rfx"
Sep 17 13:57:46.594327 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:46.594300    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:46.694935 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:46.694902    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:46.797227 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:46.797195    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:47.208273 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:47.208241    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:47.751441 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:47.751412    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3e8dc684-ddfc-4801-a7de-7874e92eb31f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4848q\" (UID: \"3e8dc684-ddfc-4801-a7de-7874e92eb31f\") " pod="openshift-insights/insights-runtime-extractor-4848q"
Sep 17 13:57:47.753685 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:47.753655    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3e8dc684-ddfc-4801-a7de-7874e92eb31f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4848q\" (UID: \"3e8dc684-ddfc-4801-a7de-7874e92eb31f\") " pod="openshift-insights/insights-runtime-extractor-4848q"
Sep 17 13:57:47.756882 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:47.756860    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-twkdq\""
Sep 17 13:57:47.764922 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:47.764902    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4848q"
Sep 17 13:57:48.053260 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:48.053227    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4848q"]
Sep 17 13:57:48.055194 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:57:48.055170    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e8dc684_ddfc_4801_a7de_7874e92eb31f.slice/crio-8c32a6d4472a4d2f6decfb1da2a8403a04dd5ef9fa564e3556d10a751dd52989 WatchSource:0}: Error finding container 8c32a6d4472a4d2f6decfb1da2a8403a04dd5ef9fa564e3556d10a751dd52989: Status 404 returned error can't find the container with id 8c32a6d4472a4d2f6decfb1da2a8403a04dd5ef9fa564e3556d10a751dd52989
Sep 17 13:57:48.074825 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:48.074795    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4848q" event={"ID":"3e8dc684-ddfc-4801-a7de-7874e92eb31f","Type":"ContainerStarted","Data":"8c32a6d4472a4d2f6decfb1da2a8403a04dd5ef9fa564e3556d10a751dd52989"}
Sep 17 13:57:49.078983 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:49.078947    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4848q" event={"ID":"3e8dc684-ddfc-4801-a7de-7874e92eb31f","Type":"ContainerStarted","Data":"d19a21373dbd7868539ac9b99aab7db8c55830c604337a4844d834a94762e1b2"}
Sep 17 13:57:50.083707 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:50.083668    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4848q" event={"ID":"3e8dc684-ddfc-4801-a7de-7874e92eb31f","Type":"ContainerStarted","Data":"6eb7a38b0e0d3f0a3c409acc42cb9be6f8e9f5d895160d8cf529be2c45b83546"}
Sep 17 13:57:50.213095 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:50.213050    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-kpx6x"]
Sep 17 13:57:50.216170 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:50.216153    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-d5l5x"]
Sep 17 13:57:50.216300 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:50.216284    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-kpx6x"
Sep 17 13:57:50.218898 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:50.218872    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 13:57:50.219017 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:50.218977    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-d5l5x"
Sep 17 13:57:50.219147 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:50.219123    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-lm5xw\""
Sep 17 13:57:50.219620 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:50.219593    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 13:57:50.221657 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:50.221633    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 17 13:57:50.221779 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:50.221677    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-6lsdh\""
Sep 17 13:57:50.233674 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:50.233653    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-d5l5x"]
Sep 17 13:57:50.234852 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:50.234829    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-kpx6x"]
Sep 17 13:57:50.235851 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:50.235826    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:50.341834 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:50.341801    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:50.371404 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:50.371384    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bxpc\" (UniqueName: \"kubernetes.io/projected/a2d73565-b83f-4582-9c80-84c460828fe6-kube-api-access-9bxpc\") pod \"downloads-76f99584f4-kpx6x\" (UID: \"a2d73565-b83f-4582-9c80-84c460828fe6\") " pod="openshift-console/downloads-76f99584f4-kpx6x"
Sep 17 13:57:50.371539 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:50.371430    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/55763440-e26d-49dd-b74f-da96ae2d99d9-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-d5l5x\" (UID: \"55763440-e26d-49dd-b74f-da96ae2d99d9\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-d5l5x"
Sep 17 13:57:50.441067 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:50.441039    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:50.471760 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:50.471735    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bxpc\" (UniqueName: \"kubernetes.io/projected/a2d73565-b83f-4582-9c80-84c460828fe6-kube-api-access-9bxpc\") pod \"downloads-76f99584f4-kpx6x\" (UID: \"a2d73565-b83f-4582-9c80-84c460828fe6\") " pod="openshift-console/downloads-76f99584f4-kpx6x"
Sep 17 13:57:50.471862 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:50.471778    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/55763440-e26d-49dd-b74f-da96ae2d99d9-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-d5l5x\" (UID: \"55763440-e26d-49dd-b74f-da96ae2d99d9\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-d5l5x"
Sep 17 13:57:50.473971 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:50.473952    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/55763440-e26d-49dd-b74f-da96ae2d99d9-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-d5l5x\" (UID: \"55763440-e26d-49dd-b74f-da96ae2d99d9\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-d5l5x"
Sep 17 13:57:50.481330 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:50.481310    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bxpc\" (UniqueName: \"kubernetes.io/projected/a2d73565-b83f-4582-9c80-84c460828fe6-kube-api-access-9bxpc\") pod \"downloads-76f99584f4-kpx6x\" (UID: \"a2d73565-b83f-4582-9c80-84c460828fe6\") " pod="openshift-console/downloads-76f99584f4-kpx6x"
Sep 17 13:57:50.528252 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:50.528196    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-kpx6x"
Sep 17 13:57:50.536250 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:50.536227    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-d5l5x"
Sep 17 13:57:50.797930 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:50.797900    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:50.813811 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:50.813704    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-kpx6x"]
Sep 17 13:57:50.832225 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:50.832117    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-d5l5x"]
Sep 17 13:57:50.876287 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:50.876228    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:50.964913 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:57:50.964879    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2d73565_b83f_4582_9c80_84c460828fe6.slice/crio-6113f8473d5200a78ce2af524fdc4e0cb25e522e834add037f4810a27c720cf3 WatchSource:0}: Error finding container 6113f8473d5200a78ce2af524fdc4e0cb25e522e834add037f4810a27c720cf3: Status 404 returned error can't find the container with id 6113f8473d5200a78ce2af524fdc4e0cb25e522e834add037f4810a27c720cf3
Sep 17 13:57:50.965639 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:57:50.965611    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55763440_e26d_49dd_b74f_da96ae2d99d9.slice/crio-ba8bd1e9249936a0bd3bf637113bb61928a0d1458c71ed527f9ca5cd07e3ba84 WatchSource:0}: Error finding container ba8bd1e9249936a0bd3bf637113bb61928a0d1458c71ed527f9ca5cd07e3ba84: Status 404 returned error can't find the container with id ba8bd1e9249936a0bd3bf637113bb61928a0d1458c71ed527f9ca5cd07e3ba84
Sep 17 13:57:51.087368 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:51.087346    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-d5l5x" event={"ID":"55763440-e26d-49dd-b74f-da96ae2d99d9","Type":"ContainerStarted","Data":"ba8bd1e9249936a0bd3bf637113bb61928a0d1458c71ed527f9ca5cd07e3ba84"}
Sep 17 13:57:51.088426 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:51.088408    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-kpx6x" event={"ID":"a2d73565-b83f-4582-9c80-84c460828fe6","Type":"ContainerStarted","Data":"6113f8473d5200a78ce2af524fdc4e0cb25e522e834add037f4810a27c720cf3"}
Sep 17 13:57:51.090248 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:51.090228    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4848q" event={"ID":"3e8dc684-ddfc-4801-a7de-7874e92eb31f","Type":"ContainerStarted","Data":"83bdaa444a70d36fa53d8e107b899c54439ae6106a056470433b0f4130e555cb"}
Sep 17 13:57:51.110095 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:51.110053    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-4848q" podStartSLOduration=33.259688035 podStartE2EDuration="36.110035397s" totalImagesPullingTime="2.850347362s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 13:57:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 13:57:51.109596875 +0000 UTC m=+140.051740521" watchObservedRunningTime="2026-09-17 13:57:51.110035397 +0000 UTC m=+140.052179044"
Sep 17 13:57:54.102946 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:54.102905    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-d5l5x" event={"ID":"55763440-e26d-49dd-b74f-da96ae2d99d9","Type":"ContainerStarted","Data":"9af9e1167ebc6a70ce74940d0c6940c040e01a2e569e425daca2bf8f2e138b18"}
Sep 17 13:57:54.103377 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:54.103118    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-d5l5x"
Sep 17 13:57:54.108748 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:54.108713    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-d5l5x"
Sep 17 13:57:54.120020 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:54.119968    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-d5l5x" podStartSLOduration=1.999310233 podStartE2EDuration="4.119951346s" totalImagesPullingTime="2.120641113s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 13:57:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 13:57:54.119602061 +0000 UTC m=+143.061745708" watchObservedRunningTime="2026-09-17 13:57:54.119951346 +0000 UTC m=+143.062094993"
Sep 17 13:57:54.819599 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:54.819560    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:54.858318 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:54.858284    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-wdgwn"]
Sep 17 13:57:54.861674 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:54.861649    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-wdgwn"
Sep 17 13:57:54.864424 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:54.864399    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 13:57:54.864424 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:54.864423    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 13:57:54.864627 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:54.864430    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 13:57:54.864627 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:54.864455    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 17 13:57:54.864627 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:54.864496    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-nxgg7\""
Sep 17 13:57:54.864627 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:54.864408    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 17 13:57:54.872490 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:54.872448    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-wdgwn"]
Sep 17 13:57:54.891370 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:54.891330    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:54.957411 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:54.957379    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:55.008744 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:55.008713    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f43dc725-01fa-4d2c-8762-5053835093a3-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-wdgwn\" (UID: \"f43dc725-01fa-4d2c-8762-5053835093a3\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-wdgwn"
Sep 17 13:57:55.008861 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:55.008761    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w7gx\" (UniqueName: \"kubernetes.io/projected/f43dc725-01fa-4d2c-8762-5053835093a3-kube-api-access-7w7gx\") pod \"prometheus-operator-68bcd7f987-wdgwn\" (UID: \"f43dc725-01fa-4d2c-8762-5053835093a3\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-wdgwn"
Sep 17 13:57:55.008933 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:55.008893    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f43dc725-01fa-4d2c-8762-5053835093a3-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-wdgwn\" (UID: \"f43dc725-01fa-4d2c-8762-5053835093a3\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-wdgwn"
Sep 17 13:57:55.008994 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:55.008954    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f43dc725-01fa-4d2c-8762-5053835093a3-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-wdgwn\" (UID: \"f43dc725-01fa-4d2c-8762-5053835093a3\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-wdgwn"
Sep 17 13:57:55.014211 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:55.014191    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-fb686c6f6-58glc"]
Sep 17 13:57:55.017308 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:55.017291    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fb686c6f6-58glc"
Sep 17 13:57:55.020423 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:55.020402    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 13:57:55.020775 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:55.020746    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 13:57:55.020844 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:55.020828    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 13:57:55.020922 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:55.020906    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-lnt98\""
Sep 17 13:57:55.020991 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:55.020976    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 13:57:55.021338 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:55.021321    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 13:57:55.031867 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:55.031845    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-fb686c6f6-58glc"]
Sep 17 13:57:55.068263 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:55.068230    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:55.109791 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:55.109736    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f43dc725-01fa-4d2c-8762-5053835093a3-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-wdgwn\" (UID: \"f43dc725-01fa-4d2c-8762-5053835093a3\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-wdgwn"
Sep 17 13:57:55.109791 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:55.109785    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f43dc725-01fa-4d2c-8762-5053835093a3-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-wdgwn\" (UID: \"f43dc725-01fa-4d2c-8762-5053835093a3\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-wdgwn"
Sep 17 13:57:55.110198 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:55.109805    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w7gx\" (UniqueName: \"kubernetes.io/projected/f43dc725-01fa-4d2c-8762-5053835093a3-kube-api-access-7w7gx\") pod \"prometheus-operator-68bcd7f987-wdgwn\" (UID: \"f43dc725-01fa-4d2c-8762-5053835093a3\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-wdgwn"
Sep 17 13:57:55.110198 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:55.109852    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f43dc725-01fa-4d2c-8762-5053835093a3-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-wdgwn\" (UID: \"f43dc725-01fa-4d2c-8762-5053835093a3\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-wdgwn"
Sep 17 13:57:55.110511 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:55.110482    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f43dc725-01fa-4d2c-8762-5053835093a3-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-wdgwn\" (UID: \"f43dc725-01fa-4d2c-8762-5053835093a3\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-wdgwn"
Sep 17 13:57:55.112560 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:55.112531    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f43dc725-01fa-4d2c-8762-5053835093a3-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-wdgwn\" (UID: \"f43dc725-01fa-4d2c-8762-5053835093a3\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-wdgwn"
Sep 17 13:57:55.112560 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:55.112538    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f43dc725-01fa-4d2c-8762-5053835093a3-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-wdgwn\" (UID: \"f43dc725-01fa-4d2c-8762-5053835093a3\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-wdgwn"
Sep 17 13:57:55.118511 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:55.118487    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w7gx\" (UniqueName: \"kubernetes.io/projected/f43dc725-01fa-4d2c-8762-5053835093a3-kube-api-access-7w7gx\") pod \"prometheus-operator-68bcd7f987-wdgwn\" (UID: \"f43dc725-01fa-4d2c-8762-5053835093a3\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-wdgwn"
Sep 17 13:57:55.172832 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:55.172807    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-wdgwn"
Sep 17 13:57:55.211221 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:55.211189    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/878f435a-160a-4634-9ed2-e91c488b6854-service-ca\") pod \"console-fb686c6f6-58glc\" (UID: \"878f435a-160a-4634-9ed2-e91c488b6854\") " pod="openshift-console/console-fb686c6f6-58glc"
Sep 17 13:57:55.211344 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:55.211236    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/878f435a-160a-4634-9ed2-e91c488b6854-console-serving-cert\") pod \"console-fb686c6f6-58glc\" (UID: \"878f435a-160a-4634-9ed2-e91c488b6854\") " pod="openshift-console/console-fb686c6f6-58glc"
Sep 17 13:57:55.211344 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:55.211261    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/878f435a-160a-4634-9ed2-e91c488b6854-oauth-serving-cert\") pod \"console-fb686c6f6-58glc\" (UID: \"878f435a-160a-4634-9ed2-e91c488b6854\") " pod="openshift-console/console-fb686c6f6-58glc"
Sep 17 13:57:55.211344 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:55.211286    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/878f435a-160a-4634-9ed2-e91c488b6854-console-oauth-config\") pod \"console-fb686c6f6-58glc\" (UID: \"878f435a-160a-4634-9ed2-e91c488b6854\") " pod="openshift-console/console-fb686c6f6-58glc"
Sep 17 13:57:55.211535 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:55.211407    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/878f435a-160a-4634-9ed2-e91c488b6854-console-config\") pod \"console-fb686c6f6-58glc\" (UID: \"878f435a-160a-4634-9ed2-e91c488b6854\") " pod="openshift-console/console-fb686c6f6-58glc"
Sep 17 13:57:55.211535 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:55.211443    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhdt5\" (UniqueName: \"kubernetes.io/projected/878f435a-160a-4634-9ed2-e91c488b6854-kube-api-access-jhdt5\") pod \"console-fb686c6f6-58glc\" (UID: \"878f435a-160a-4634-9ed2-e91c488b6854\") " pod="openshift-console/console-fb686c6f6-58glc"
Sep 17 13:57:55.312338 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:55.312310    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/878f435a-160a-4634-9ed2-e91c488b6854-console-config\") pod \"console-fb686c6f6-58glc\" (UID: \"878f435a-160a-4634-9ed2-e91c488b6854\") " pod="openshift-console/console-fb686c6f6-58glc"
Sep 17 13:57:55.312506 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:55.312344    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhdt5\" (UniqueName: \"kubernetes.io/projected/878f435a-160a-4634-9ed2-e91c488b6854-kube-api-access-jhdt5\") pod \"console-fb686c6f6-58glc\" (UID: \"878f435a-160a-4634-9ed2-e91c488b6854\") " pod="openshift-console/console-fb686c6f6-58glc"
Sep 17 13:57:55.312506 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:55.312381    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/878f435a-160a-4634-9ed2-e91c488b6854-service-ca\") pod \"console-fb686c6f6-58glc\" (UID: \"878f435a-160a-4634-9ed2-e91c488b6854\") " pod="openshift-console/console-fb686c6f6-58glc"
Sep 17 13:57:55.312633 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:55.312502    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/878f435a-160a-4634-9ed2-e91c488b6854-console-serving-cert\") pod \"console-fb686c6f6-58glc\" (UID: \"878f435a-160a-4634-9ed2-e91c488b6854\") " pod="openshift-console/console-fb686c6f6-58glc"
Sep 17 13:57:55.312633 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:55.312541    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/878f435a-160a-4634-9ed2-e91c488b6854-oauth-serving-cert\") pod \"console-fb686c6f6-58glc\" (UID: \"878f435a-160a-4634-9ed2-e91c488b6854\") " pod="openshift-console/console-fb686c6f6-58glc"
Sep 17 13:57:55.312633 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:55.312575    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/878f435a-160a-4634-9ed2-e91c488b6854-console-oauth-config\") pod \"console-fb686c6f6-58glc\" (UID: \"878f435a-160a-4634-9ed2-e91c488b6854\") " pod="openshift-console/console-fb686c6f6-58glc"
Sep 17 13:57:55.313158 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:55.313129    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/878f435a-160a-4634-9ed2-e91c488b6854-console-config\") pod \"console-fb686c6f6-58glc\" (UID: \"878f435a-160a-4634-9ed2-e91c488b6854\") " pod="openshift-console/console-fb686c6f6-58glc"
Sep 17 13:57:55.313158 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:55.313138    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/878f435a-160a-4634-9ed2-e91c488b6854-service-ca\") pod \"console-fb686c6f6-58glc\" (UID: \"878f435a-160a-4634-9ed2-e91c488b6854\") " pod="openshift-console/console-fb686c6f6-58glc"
Sep 17 13:57:55.313324 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:55.313192    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/878f435a-160a-4634-9ed2-e91c488b6854-oauth-serving-cert\") pod \"console-fb686c6f6-58glc\" (UID: \"878f435a-160a-4634-9ed2-e91c488b6854\") " pod="openshift-console/console-fb686c6f6-58glc"
Sep 17 13:57:55.315063 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:55.315039    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/878f435a-160a-4634-9ed2-e91c488b6854-console-oauth-config\") pod \"console-fb686c6f6-58glc\" (UID: \"878f435a-160a-4634-9ed2-e91c488b6854\") " pod="openshift-console/console-fb686c6f6-58glc"
Sep 17 13:57:55.315244 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:55.315225    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/878f435a-160a-4634-9ed2-e91c488b6854-console-serving-cert\") pod \"console-fb686c6f6-58glc\" (UID: \"878f435a-160a-4634-9ed2-e91c488b6854\") " pod="openshift-console/console-fb686c6f6-58glc"
Sep 17 13:57:55.325077 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:55.325059    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhdt5\" (UniqueName: \"kubernetes.io/projected/878f435a-160a-4634-9ed2-e91c488b6854-kube-api-access-jhdt5\") pod \"console-fb686c6f6-58glc\" (UID: \"878f435a-160a-4634-9ed2-e91c488b6854\") " pod="openshift-console/console-fb686c6f6-58glc"
Sep 17 13:57:55.327866 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:55.327840    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fb686c6f6-58glc"
Sep 17 13:57:55.460722 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:55.460692    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-wdgwn"]
Sep 17 13:57:55.462657 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:57:55.462625    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf43dc725_01fa_4d2c_8762_5053835093a3.slice/crio-bd2ef17a4d2d42c6396396f964db387f1e714238c44bb959c663ed1880273019 WatchSource:0}: Error finding container bd2ef17a4d2d42c6396396f964db387f1e714238c44bb959c663ed1880273019: Status 404 returned error can't find the container with id bd2ef17a4d2d42c6396396f964db387f1e714238c44bb959c663ed1880273019
Sep 17 13:57:55.610282 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:55.610176    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-fb686c6f6-58glc"]
Sep 17 13:57:55.613014 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:57:55.612987    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod878f435a_160a_4634_9ed2_e91c488b6854.slice/crio-c1360be142aa0ef729e3fd7fb57edae82a314190b702e99e897b94b93b73fa8e WatchSource:0}: Error finding container c1360be142aa0ef729e3fd7fb57edae82a314190b702e99e897b94b93b73fa8e: Status 404 returned error can't find the container with id c1360be142aa0ef729e3fd7fb57edae82a314190b702e99e897b94b93b73fa8e
Sep 17 13:57:55.616428 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:55.616399    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:55.716879 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:55.716798    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:55.819424 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:55.819393    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:55.919899 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:55.919858    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:56.112123 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:56.112087    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fb686c6f6-58glc" event={"ID":"878f435a-160a-4634-9ed2-e91c488b6854","Type":"ContainerStarted","Data":"c1360be142aa0ef729e3fd7fb57edae82a314190b702e99e897b94b93b73fa8e"}
Sep 17 13:57:56.113585 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:56.113558    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-wdgwn" event={"ID":"f43dc725-01fa-4d2c-8762-5053835093a3","Type":"ContainerStarted","Data":"bd2ef17a4d2d42c6396396f964db387f1e714238c44bb959c663ed1880273019"}
Sep 17 13:57:57.121089 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:57.121049    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-wdgwn" event={"ID":"f43dc725-01fa-4d2c-8762-5053835093a3","Type":"ContainerStarted","Data":"30991f3a71f84eb06c138e169efb8d77cd17cc26e10dd14dd86d5c98c2eb4b50"}
Sep 17 13:57:57.121484 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:57.121110    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-wdgwn" event={"ID":"f43dc725-01fa-4d2c-8762-5053835093a3","Type":"ContainerStarted","Data":"ab4680b1d0d41a48724e361f9bfde8e859609562097d837ba3a7710bfa654814"}
Sep 17 13:57:57.149711 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:57.149656    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-wdgwn" podStartSLOduration=1.707739422 podStartE2EDuration="3.149637083s" totalImagesPullingTime="1.441897661s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 13:57:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 13:57:57.148209773 +0000 UTC m=+146.090353417" watchObservedRunningTime="2026-09-17 13:57:57.149637083 +0000 UTC m=+146.091780731"
Sep 17 13:57:57.337631 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:57.337597    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:59.013531 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.013500    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:59.130711 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.130672    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fb686c6f6-58glc" event={"ID":"878f435a-160a-4634-9ed2-e91c488b6854","Type":"ContainerStarted","Data":"3d4032128f4155eb92089043d30cd830f0b15be0400551ba2125c133fc52d41a"}
Sep 17 13:57:59.235785 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.235700    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:59.297987 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.297931    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-fb686c6f6-58glc" podStartSLOduration=2.018958891 podStartE2EDuration="5.297908459s" totalImagesPullingTime="3.278949568s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 13:57:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 13:57:59.150261223 +0000 UTC m=+148.092404868" watchObservedRunningTime="2026-09-17 13:57:59.297908459 +0000 UTC m=+148.240052105"
Sep 17 13:57:59.298651 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.298622    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-blrzn"]
Sep 17 13:57:59.302275 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.302256    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-blrzn"
Sep 17 13:57:59.304633 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.304613    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-7btsj\""
Sep 17 13:57:59.304730 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.304716    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 17 13:57:59.305128 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.305104    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 17 13:57:59.314502 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.314481    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-blrzn"]
Sep 17 13:57:59.322112 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.322034    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:59.323036 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.323018    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-gqkh8"]
Sep 17 13:57:59.327128 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.327109    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-sdzrz"]
Sep 17 13:57:59.328367 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.327236    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gqkh8"
Sep 17 13:57:59.329683 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.329662    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 13:57:59.330315 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.330295    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 13:57:59.334073 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.330948    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sdzrz"
Sep 17 13:57:59.334073 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.331037    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 13:57:59.334073 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.331222    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-s544l\""
Sep 17 13:57:59.335367 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.334640    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 17 13:57:59.335367 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.335157    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 17 13:57:59.335367 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.335267    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 17 13:57:59.335367 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.335358    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-cxpz6\""
Sep 17 13:57:59.343810 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.343785    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7ae12eb6-f825-4313-8884-1e879544e4f6-metrics-client-ca\") pod \"node-exporter-gqkh8\" (UID: \"7ae12eb6-f825-4313-8884-1e879544e4f6\") " pod="openshift-monitoring/node-exporter-gqkh8"
Sep 17 13:57:59.343907 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.343817    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz8wk\" (UniqueName: \"kubernetes.io/projected/b89d8a9d-a0d5-4578-bf8a-1711c9e5646a-kube-api-access-kz8wk\") pod \"openshift-state-metrics-67dcf8c979-blrzn\" (UID: \"b89d8a9d-a0d5-4578-bf8a-1711c9e5646a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-blrzn"
Sep 17 13:57:59.343907 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.343836    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a35f644c-4064-4099-8721-d796bddb0644-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-sdzrz\" (UID: \"a35f644c-4064-4099-8721-d796bddb0644\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sdzrz"
Sep 17 13:57:59.343907 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.343856    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7ae12eb6-f825-4313-8884-1e879544e4f6-node-exporter-textfile\") pod \"node-exporter-gqkh8\" (UID: \"7ae12eb6-f825-4313-8884-1e879544e4f6\") " pod="openshift-monitoring/node-exporter-gqkh8"
Sep 17 13:57:59.343907 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.343889    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7ae12eb6-f825-4313-8884-1e879544e4f6-node-exporter-accelerators-collector-config\") pod \"node-exporter-gqkh8\" (UID: \"7ae12eb6-f825-4313-8884-1e879544e4f6\") " pod="openshift-monitoring/node-exporter-gqkh8"
Sep 17 13:57:59.344101 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.343914    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b89d8a9d-a0d5-4578-bf8a-1711c9e5646a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-blrzn\" (UID: \"b89d8a9d-a0d5-4578-bf8a-1711c9e5646a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-blrzn"
Sep 17 13:57:59.344101 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.343939    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7ae12eb6-f825-4313-8884-1e879544e4f6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gqkh8\" (UID: \"7ae12eb6-f825-4313-8884-1e879544e4f6\") " pod="openshift-monitoring/node-exporter-gqkh8"
Sep 17 13:57:59.344101 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.343963    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7ae12eb6-f825-4313-8884-1e879544e4f6-root\") pod \"node-exporter-gqkh8\" (UID: \"7ae12eb6-f825-4313-8884-1e879544e4f6\") " pod="openshift-monitoring/node-exporter-gqkh8"
Sep 17 13:57:59.344101 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.344021    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7ae12eb6-f825-4313-8884-1e879544e4f6-sys\") pod \"node-exporter-gqkh8\" (UID: \"7ae12eb6-f825-4313-8884-1e879544e4f6\") " pod="openshift-monitoring/node-exporter-gqkh8"
Sep 17 13:57:59.344101 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.344082    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a35f644c-4064-4099-8721-d796bddb0644-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-sdzrz\" (UID: \"a35f644c-4064-4099-8721-d796bddb0644\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sdzrz"
Sep 17 13:57:59.344350 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.344131    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8498\" (UniqueName: \"kubernetes.io/projected/7ae12eb6-f825-4313-8884-1e879544e4f6-kube-api-access-h8498\") pod \"node-exporter-gqkh8\" (UID: \"7ae12eb6-f825-4313-8884-1e879544e4f6\") " pod="openshift-monitoring/node-exporter-gqkh8"
Sep 17 13:57:59.344350 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.344173    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7ae12eb6-f825-4313-8884-1e879544e4f6-node-exporter-wtmp\") pod \"node-exporter-gqkh8\" (UID: \"7ae12eb6-f825-4313-8884-1e879544e4f6\") " pod="openshift-monitoring/node-exporter-gqkh8"
Sep 17 13:57:59.344350 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.344209    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a35f644c-4064-4099-8721-d796bddb0644-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-sdzrz\" (UID: \"a35f644c-4064-4099-8721-d796bddb0644\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sdzrz"
Sep 17 13:57:59.344350 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.344226    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a35f644c-4064-4099-8721-d796bddb0644-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-sdzrz\" (UID: \"a35f644c-4064-4099-8721-d796bddb0644\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sdzrz"
Sep 17 13:57:59.344350 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.344270    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a35f644c-4064-4099-8721-d796bddb0644-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-sdzrz\" (UID: \"a35f644c-4064-4099-8721-d796bddb0644\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sdzrz"
Sep 17 13:57:59.344350 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.344312    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b89d8a9d-a0d5-4578-bf8a-1711c9e5646a-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-blrzn\" (UID: \"b89d8a9d-a0d5-4578-bf8a-1711c9e5646a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-blrzn"
Sep 17 13:57:59.344656 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.344372    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b89d8a9d-a0d5-4578-bf8a-1711c9e5646a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-blrzn\" (UID: \"b89d8a9d-a0d5-4578-bf8a-1711c9e5646a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-blrzn"
Sep 17 13:57:59.344656 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.344400    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7ae12eb6-f825-4313-8884-1e879544e4f6-node-exporter-tls\") pod \"node-exporter-gqkh8\" (UID: \"7ae12eb6-f825-4313-8884-1e879544e4f6\") " pod="openshift-monitoring/node-exporter-gqkh8"
Sep 17 13:57:59.344656 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.344422    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kwvm\" (UniqueName: \"kubernetes.io/projected/a35f644c-4064-4099-8721-d796bddb0644-kube-api-access-5kwvm\") pod \"kube-state-metrics-59fbdd77d-sdzrz\" (UID: \"a35f644c-4064-4099-8721-d796bddb0644\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sdzrz"
Sep 17 13:57:59.347278 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.347256    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-sdzrz"]
Sep 17 13:57:59.445228 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.445202    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7ae12eb6-f825-4313-8884-1e879544e4f6-node-exporter-wtmp\") pod \"node-exporter-gqkh8\" (UID: \"7ae12eb6-f825-4313-8884-1e879544e4f6\") " pod="openshift-monitoring/node-exporter-gqkh8"
Sep 17 13:57:59.445228 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.445234    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a35f644c-4064-4099-8721-d796bddb0644-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-sdzrz\" (UID: \"a35f644c-4064-4099-8721-d796bddb0644\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sdzrz"
Sep 17 13:57:59.445491 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.445291    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a35f644c-4064-4099-8721-d796bddb0644-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-sdzrz\" (UID: \"a35f644c-4064-4099-8721-d796bddb0644\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sdzrz"
Sep 17 13:57:59.445491 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.445343    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a35f644c-4064-4099-8721-d796bddb0644-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-sdzrz\" (UID: \"a35f644c-4064-4099-8721-d796bddb0644\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sdzrz"
Sep 17 13:57:59.445491 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.445376    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b89d8a9d-a0d5-4578-bf8a-1711c9e5646a-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-blrzn\" (UID: \"b89d8a9d-a0d5-4578-bf8a-1711c9e5646a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-blrzn"
Sep 17 13:57:59.445491 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.445400    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7ae12eb6-f825-4313-8884-1e879544e4f6-node-exporter-wtmp\") pod \"node-exporter-gqkh8\" (UID: \"7ae12eb6-f825-4313-8884-1e879544e4f6\") " pod="openshift-monitoring/node-exporter-gqkh8"
Sep 17 13:57:59.445491 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.445409    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b89d8a9d-a0d5-4578-bf8a-1711c9e5646a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-blrzn\" (UID: \"b89d8a9d-a0d5-4578-bf8a-1711c9e5646a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-blrzn"
Sep 17 13:57:59.445491 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.445454    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7ae12eb6-f825-4313-8884-1e879544e4f6-node-exporter-tls\") pod \"node-exporter-gqkh8\" (UID: \"7ae12eb6-f825-4313-8884-1e879544e4f6\") " pod="openshift-monitoring/node-exporter-gqkh8"
Sep 17 13:57:59.445801 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.445505    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kwvm\" (UniqueName: \"kubernetes.io/projected/a35f644c-4064-4099-8721-d796bddb0644-kube-api-access-5kwvm\") pod \"kube-state-metrics-59fbdd77d-sdzrz\" (UID: \"a35f644c-4064-4099-8721-d796bddb0644\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sdzrz"
Sep 17 13:57:59.445801 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.445535    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7ae12eb6-f825-4313-8884-1e879544e4f6-metrics-client-ca\") pod \"node-exporter-gqkh8\" (UID: \"7ae12eb6-f825-4313-8884-1e879544e4f6\") " pod="openshift-monitoring/node-exporter-gqkh8"
Sep 17 13:57:59.445801 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.445563    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz8wk\" (UniqueName: \"kubernetes.io/projected/b89d8a9d-a0d5-4578-bf8a-1711c9e5646a-kube-api-access-kz8wk\") pod \"openshift-state-metrics-67dcf8c979-blrzn\" (UID: \"b89d8a9d-a0d5-4578-bf8a-1711c9e5646a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-blrzn"
Sep 17 13:57:59.445801 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.445592    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a35f644c-4064-4099-8721-d796bddb0644-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-sdzrz\" (UID: \"a35f644c-4064-4099-8721-d796bddb0644\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sdzrz"
Sep 17 13:57:59.445801 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.445621    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7ae12eb6-f825-4313-8884-1e879544e4f6-node-exporter-textfile\") pod \"node-exporter-gqkh8\" (UID: \"7ae12eb6-f825-4313-8884-1e879544e4f6\") " pod="openshift-monitoring/node-exporter-gqkh8"
Sep 17 13:57:59.445801 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.445659    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7ae12eb6-f825-4313-8884-1e879544e4f6-node-exporter-accelerators-collector-config\") pod \"node-exporter-gqkh8\" (UID: \"7ae12eb6-f825-4313-8884-1e879544e4f6\") " pod="openshift-monitoring/node-exporter-gqkh8"
Sep 17 13:57:59.445801 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.445665    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a35f644c-4064-4099-8721-d796bddb0644-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-sdzrz\" (UID: \"a35f644c-4064-4099-8721-d796bddb0644\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sdzrz"
Sep 17 13:57:59.445801 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.445685    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b89d8a9d-a0d5-4578-bf8a-1711c9e5646a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-blrzn\" (UID: \"b89d8a9d-a0d5-4578-bf8a-1711c9e5646a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-blrzn"
Sep 17 13:57:59.445801 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.445712    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7ae12eb6-f825-4313-8884-1e879544e4f6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gqkh8\" (UID: \"7ae12eb6-f825-4313-8884-1e879544e4f6\") " pod="openshift-monitoring/node-exporter-gqkh8"
Sep 17 13:57:59.445801 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.445737    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7ae12eb6-f825-4313-8884-1e879544e4f6-root\") pod \"node-exporter-gqkh8\" (UID: \"7ae12eb6-f825-4313-8884-1e879544e4f6\") " pod="openshift-monitoring/node-exporter-gqkh8"
Sep 17 13:57:59.445801 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.445786    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7ae12eb6-f825-4313-8884-1e879544e4f6-sys\") pod \"node-exporter-gqkh8\" (UID: \"7ae12eb6-f825-4313-8884-1e879544e4f6\") " pod="openshift-monitoring/node-exporter-gqkh8"
Sep 17 13:57:59.446291 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.445819    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a35f644c-4064-4099-8721-d796bddb0644-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-sdzrz\" (UID: \"a35f644c-4064-4099-8721-d796bddb0644\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sdzrz"
Sep 17 13:57:59.446291 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.445865    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8498\" (UniqueName: \"kubernetes.io/projected/7ae12eb6-f825-4313-8884-1e879544e4f6-kube-api-access-h8498\") pod \"node-exporter-gqkh8\" (UID: \"7ae12eb6-f825-4313-8884-1e879544e4f6\") " pod="openshift-monitoring/node-exporter-gqkh8"
Sep 17 13:57:59.446291 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:57:59.445993    2560 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 17 13:57:59.446291 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:57:59.446048    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ae12eb6-f825-4313-8884-1e879544e4f6-node-exporter-tls podName:7ae12eb6-f825-4313-8884-1e879544e4f6 nodeName:}" failed. No retries permitted until 2026-09-17 13:57:59.946031683 +0000 UTC m=+148.888175310 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/7ae12eb6-f825-4313-8884-1e879544e4f6-node-exporter-tls") pod "node-exporter-gqkh8" (UID: "7ae12eb6-f825-4313-8884-1e879544e4f6") : secret "node-exporter-tls" not found
Sep 17 13:57:59.446291 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.446130    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a35f644c-4064-4099-8721-d796bddb0644-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-sdzrz\" (UID: \"a35f644c-4064-4099-8721-d796bddb0644\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sdzrz"
Sep 17 13:57:59.446291 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:57:59.446145    2560 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 17 13:57:59.446291 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:57:59.446200    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a35f644c-4064-4099-8721-d796bddb0644-kube-state-metrics-tls podName:a35f644c-4064-4099-8721-d796bddb0644 nodeName:}" failed. No retries permitted until 2026-09-17 13:57:59.946183708 +0000 UTC m=+148.888327333 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/a35f644c-4064-4099-8721-d796bddb0644-kube-state-metrics-tls") pod "kube-state-metrics-59fbdd77d-sdzrz" (UID: "a35f644c-4064-4099-8721-d796bddb0644") : secret "kube-state-metrics-tls" not found
Sep 17 13:57:59.446731 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.446405    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7ae12eb6-f825-4313-8884-1e879544e4f6-node-exporter-textfile\") pod \"node-exporter-gqkh8\" (UID: \"7ae12eb6-f825-4313-8884-1e879544e4f6\") " pod="openshift-monitoring/node-exporter-gqkh8"
Sep 17 13:57:59.446731 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:57:59.446476    2560 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 17 13:57:59.446731 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.446486    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7ae12eb6-f825-4313-8884-1e879544e4f6-root\") pod \"node-exporter-gqkh8\" (UID: \"7ae12eb6-f825-4313-8884-1e879544e4f6\") " pod="openshift-monitoring/node-exporter-gqkh8"
Sep 17 13:57:59.446731 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:57:59.446524    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b89d8a9d-a0d5-4578-bf8a-1711c9e5646a-openshift-state-metrics-tls podName:b89d8a9d-a0d5-4578-bf8a-1711c9e5646a nodeName:}" failed. No retries permitted until 2026-09-17 13:57:59.946509013 +0000 UTC m=+148.888652642 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/b89d8a9d-a0d5-4578-bf8a-1711c9e5646a-openshift-state-metrics-tls") pod "openshift-state-metrics-67dcf8c979-blrzn" (UID: "b89d8a9d-a0d5-4578-bf8a-1711c9e5646a") : secret "openshift-state-metrics-tls" not found
Sep 17 13:57:59.446731 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.446532    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7ae12eb6-f825-4313-8884-1e879544e4f6-sys\") pod \"node-exporter-gqkh8\" (UID: \"7ae12eb6-f825-4313-8884-1e879544e4f6\") " pod="openshift-monitoring/node-exporter-gqkh8"
Sep 17 13:57:59.446967 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.446769    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7ae12eb6-f825-4313-8884-1e879544e4f6-metrics-client-ca\") pod \"node-exporter-gqkh8\" (UID: \"7ae12eb6-f825-4313-8884-1e879544e4f6\") " pod="openshift-monitoring/node-exporter-gqkh8"
Sep 17 13:57:59.446967 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.446786    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b89d8a9d-a0d5-4578-bf8a-1711c9e5646a-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-blrzn\" (UID: \"b89d8a9d-a0d5-4578-bf8a-1711c9e5646a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-blrzn"
Sep 17 13:57:59.447176 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.447124    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a35f644c-4064-4099-8721-d796bddb0644-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-sdzrz\" (UID: \"a35f644c-4064-4099-8721-d796bddb0644\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sdzrz"
Sep 17 13:57:59.447247 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.447191    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7ae12eb6-f825-4313-8884-1e879544e4f6-node-exporter-accelerators-collector-config\") pod \"node-exporter-gqkh8\" (UID: \"7ae12eb6-f825-4313-8884-1e879544e4f6\") " pod="openshift-monitoring/node-exporter-gqkh8"
Sep 17 13:57:59.448105 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.448084    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b89d8a9d-a0d5-4578-bf8a-1711c9e5646a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-blrzn\" (UID: \"b89d8a9d-a0d5-4578-bf8a-1711c9e5646a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-blrzn"
Sep 17 13:57:59.448238 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.448216    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a35f644c-4064-4099-8721-d796bddb0644-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-sdzrz\" (UID: \"a35f644c-4064-4099-8721-d796bddb0644\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sdzrz"
Sep 17 13:57:59.448604 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.448580    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7ae12eb6-f825-4313-8884-1e879544e4f6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gqkh8\" (UID: \"7ae12eb6-f825-4313-8884-1e879544e4f6\") " pod="openshift-monitoring/node-exporter-gqkh8"
Sep 17 13:57:59.455753 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.455732    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz8wk\" (UniqueName: \"kubernetes.io/projected/b89d8a9d-a0d5-4578-bf8a-1711c9e5646a-kube-api-access-kz8wk\") pod \"openshift-state-metrics-67dcf8c979-blrzn\" (UID: \"b89d8a9d-a0d5-4578-bf8a-1711c9e5646a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-blrzn"
Sep 17 13:57:59.455841 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.455767    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8498\" (UniqueName: \"kubernetes.io/projected/7ae12eb6-f825-4313-8884-1e879544e4f6-kube-api-access-h8498\") pod \"node-exporter-gqkh8\" (UID: \"7ae12eb6-f825-4313-8884-1e879544e4f6\") " pod="openshift-monitoring/node-exporter-gqkh8"
Sep 17 13:57:59.459727 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.459703    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kwvm\" (UniqueName: \"kubernetes.io/projected/a35f644c-4064-4099-8721-d796bddb0644-kube-api-access-5kwvm\") pod \"kube-state-metrics-59fbdd77d-sdzrz\" (UID: \"a35f644c-4064-4099-8721-d796bddb0644\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sdzrz"
Sep 17 13:57:59.542361 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.542291    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:59.669796 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.669764    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:59.778033 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.777989    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:57:59.950845 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.950815    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7ae12eb6-f825-4313-8884-1e879544e4f6-node-exporter-tls\") pod \"node-exporter-gqkh8\" (UID: \"7ae12eb6-f825-4313-8884-1e879544e4f6\") " pod="openshift-monitoring/node-exporter-gqkh8"
Sep 17 13:57:59.951120 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.950865    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a35f644c-4064-4099-8721-d796bddb0644-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-sdzrz\" (UID: \"a35f644c-4064-4099-8721-d796bddb0644\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sdzrz"
Sep 17 13:57:59.951120 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.950899    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b89d8a9d-a0d5-4578-bf8a-1711c9e5646a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-blrzn\" (UID: \"b89d8a9d-a0d5-4578-bf8a-1711c9e5646a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-blrzn"
Sep 17 13:57:59.951120 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:57:59.951035    2560 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 17 13:57:59.951120 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:57:59.951098    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a35f644c-4064-4099-8721-d796bddb0644-kube-state-metrics-tls podName:a35f644c-4064-4099-8721-d796bddb0644 nodeName:}" failed. No retries permitted until 2026-09-17 13:58:00.951080131 +0000 UTC m=+149.893223763 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/a35f644c-4064-4099-8721-d796bddb0644-kube-state-metrics-tls") pod "kube-state-metrics-59fbdd77d-sdzrz" (UID: "a35f644c-4064-4099-8721-d796bddb0644") : secret "kube-state-metrics-tls" not found
Sep 17 13:57:59.953411 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.953386    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7ae12eb6-f825-4313-8884-1e879544e4f6-node-exporter-tls\") pod \"node-exporter-gqkh8\" (UID: \"7ae12eb6-f825-4313-8884-1e879544e4f6\") " pod="openshift-monitoring/node-exporter-gqkh8"
Sep 17 13:57:59.953549 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:57:59.953499    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b89d8a9d-a0d5-4578-bf8a-1711c9e5646a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-blrzn\" (UID: \"b89d8a9d-a0d5-4578-bf8a-1711c9e5646a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-blrzn"
Sep 17 13:58:00.150181 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:00.150144    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:58:00.213210 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:00.213145    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-blrzn"
Sep 17 13:58:00.240574 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:00.240550    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gqkh8"
Sep 17 13:58:00.311159 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:00.311127    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:58:00.416818 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:00.416781    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:58:00.517270 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:00.517157    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:58:00.525056 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:00.525014    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 13:58:00.529025 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:00.529000    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:58:00.531376 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:00.531351    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 17 13:58:00.533077 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:00.533034    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 17 13:58:00.533077 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:00.533044    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 17 13:58:00.533439 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:00.533285    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 17 13:58:00.533439 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:00.533326    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-cqm8j\""
Sep 17 13:58:00.533439 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:00.533346    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 17 13:58:00.533439 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:00.533359    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 17 13:58:00.533439 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:00.533320    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 17 13:58:00.533439 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:00.533397    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 17 13:58:00.536360 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:00.536322    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 17 13:58:00.545987 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:00.545955    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 13:58:00.556584 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:00.556561    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a29762f9-e0d5-43cd-b19f-2db732145b53-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a29762f9-e0d5-43cd-b19f-2db732145b53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:58:00.556689 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:00.556594    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a29762f9-e0d5-43cd-b19f-2db732145b53-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a29762f9-e0d5-43cd-b19f-2db732145b53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:58:00.556689 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:00.556668    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a29762f9-e0d5-43cd-b19f-2db732145b53-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a29762f9-e0d5-43cd-b19f-2db732145b53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:58:00.556799 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:00.556737    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a29762f9-e0d5-43cd-b19f-2db732145b53-config-volume\") pod \"alertmanager-main-0\" (UID: \"a29762f9-e0d5-43cd-b19f-2db732145b53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:58:00.556799 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:00.556769    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a29762f9-e0d5-43cd-b19f-2db732145b53-config-out\") pod \"alertmanager-main-0\" (UID: \"a29762f9-e0d5-43cd-b19f-2db732145b53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:58:00.556904 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:00.556802    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a29762f9-e0d5-43cd-b19f-2db732145b53-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a29762f9-e0d5-43cd-b19f-2db732145b53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:58:00.556904 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:00.556825    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a29762f9-e0d5-43cd-b19f-2db732145b53-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a29762f9-e0d5-43cd-b19f-2db732145b53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:58:00.556904 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:00.556853    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lnq4\" (UniqueName: \"kubernetes.io/projected/a29762f9-e0d5-43cd-b19f-2db732145b53-kube-api-access-2lnq4\") pod \"alertmanager-main-0\" (UID: \"a29762f9-e0d5-43cd-b19f-2db732145b53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:58:00.556904 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:00.556878    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a29762f9-e0d5-43cd-b19f-2db732145b53-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a29762f9-e0d5-43cd-b19f-2db732145b53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:58:00.557066 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:00.556908    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a29762f9-e0d5-43cd-b19f-2db732145b53-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a29762f9-e0d5-43cd-b19f-2db732145b53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:58:00.557066 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:00.556947    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a29762f9-e0d5-43cd-b19f-2db732145b53-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a29762f9-e0d5-43cd-b19f-2db732145b53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:58:00.557066 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:00.556970    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a29762f9-e0d5-43cd-b19f-2db732145b53-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a29762f9-e0d5-43cd-b19f-2db732145b53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:58:00.557066 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:00.557029    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a29762f9-e0d5-43cd-b19f-2db732145b53-web-config\") pod \"alertmanager-main-0\" (UID: \"a29762f9-e0d5-43cd-b19f-2db732145b53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:58:00.657813 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:00.657773    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a29762f9-e0d5-43cd-b19f-2db732145b53-config-volume\") pod \"alertmanager-main-0\" (UID: \"a29762f9-e0d5-43cd-b19f-2db732145b53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:58:00.657813 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:00.657818    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a29762f9-e0d5-43cd-b19f-2db732145b53-config-out\") pod \"alertmanager-main-0\" (UID: \"a29762f9-e0d5-43cd-b19f-2db732145b53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:58:00.658037 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:00.657848    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a29762f9-e0d5-43cd-b19f-2db732145b53-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a29762f9-e0d5-43cd-b19f-2db732145b53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:58:00.658037 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:00.658016    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a29762f9-e0d5-43cd-b19f-2db732145b53-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a29762f9-e0d5-43cd-b19f-2db732145b53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:58:00.658141 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:00.658061    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lnq4\" (UniqueName: \"kubernetes.io/projected/a29762f9-e0d5-43cd-b19f-2db732145b53-kube-api-access-2lnq4\") pod \"alertmanager-main-0\" (UID: \"a29762f9-e0d5-43cd-b19f-2db732145b53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:58:00.658141 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:00.658091    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a29762f9-e0d5-43cd-b19f-2db732145b53-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a29762f9-e0d5-43cd-b19f-2db732145b53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:58:00.658141 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:00.658128    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a29762f9-e0d5-43cd-b19f-2db732145b53-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a29762f9-e0d5-43cd-b19f-2db732145b53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:58:00.658298 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:00.658172    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a29762f9-e0d5-43cd-b19f-2db732145b53-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a29762f9-e0d5-43cd-b19f-2db732145b53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:58:00.658298 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:00.658194    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a29762f9-e0d5-43cd-b19f-2db732145b53-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a29762f9-e0d5-43cd-b19f-2db732145b53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:58:00.658298 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:00.658235    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a29762f9-e0d5-43cd-b19f-2db732145b53-web-config\") pod \"alertmanager-main-0\" (UID: \"a29762f9-e0d5-43cd-b19f-2db732145b53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:58:00.660175 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:00.658878    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a29762f9-e0d5-43cd-b19f-2db732145b53-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a29762f9-e0d5-43cd-b19f-2db732145b53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:58:00.660175 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:00.659622    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a29762f9-e0d5-43cd-b19f-2db732145b53-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a29762f9-e0d5-43cd-b19f-2db732145b53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:58:00.660175 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:00.659674    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a29762f9-e0d5-43cd-b19f-2db732145b53-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a29762f9-e0d5-43cd-b19f-2db732145b53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:58:00.660175 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:00.659711    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a29762f9-e0d5-43cd-b19f-2db732145b53-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a29762f9-e0d5-43cd-b19f-2db732145b53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:58:00.660175 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:00.659764    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a29762f9-e0d5-43cd-b19f-2db732145b53-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a29762f9-e0d5-43cd-b19f-2db732145b53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:58:00.661529 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:00.660940    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a29762f9-e0d5-43cd-b19f-2db732145b53-config-volume\") pod \"alertmanager-main-0\" (UID: \"a29762f9-e0d5-43cd-b19f-2db732145b53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:58:00.661529 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:00.661016    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a29762f9-e0d5-43cd-b19f-2db732145b53-config-out\") pod \"alertmanager-main-0\" (UID: \"a29762f9-e0d5-43cd-b19f-2db732145b53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:58:00.661529 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:00.661018    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a29762f9-e0d5-43cd-b19f-2db732145b53-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a29762f9-e0d5-43cd-b19f-2db732145b53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:58:00.661529 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:58:00.661040    2560 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 17 13:58:00.661529 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:00.661023    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a29762f9-e0d5-43cd-b19f-2db732145b53-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a29762f9-e0d5-43cd-b19f-2db732145b53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:58:00.661529 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:58:00.661106    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a29762f9-e0d5-43cd-b19f-2db732145b53-secret-alertmanager-main-tls podName:a29762f9-e0d5-43cd-b19f-2db732145b53 nodeName:}" failed. No retries permitted until 2026-09-17 13:58:01.16108765 +0000 UTC m=+150.103231291 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/a29762f9-e0d5-43cd-b19f-2db732145b53-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "a29762f9-e0d5-43cd-b19f-2db732145b53") : secret "alertmanager-main-tls" not found
Sep 17 13:58:00.661529 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:00.661306    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a29762f9-e0d5-43cd-b19f-2db732145b53-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a29762f9-e0d5-43cd-b19f-2db732145b53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:58:00.661938 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:00.661532    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a29762f9-e0d5-43cd-b19f-2db732145b53-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a29762f9-e0d5-43cd-b19f-2db732145b53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:58:00.661938 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:00.661875    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a29762f9-e0d5-43cd-b19f-2db732145b53-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a29762f9-e0d5-43cd-b19f-2db732145b53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:58:00.662035 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:00.661940    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a29762f9-e0d5-43cd-b19f-2db732145b53-web-config\") pod \"alertmanager-main-0\" (UID: \"a29762f9-e0d5-43cd-b19f-2db732145b53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:58:00.662745 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:00.662724    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a29762f9-e0d5-43cd-b19f-2db732145b53-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a29762f9-e0d5-43cd-b19f-2db732145b53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:58:00.672884 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:00.672852    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lnq4\" (UniqueName: \"kubernetes.io/projected/a29762f9-e0d5-43cd-b19f-2db732145b53-kube-api-access-2lnq4\") pod \"alertmanager-main-0\" (UID: \"a29762f9-e0d5-43cd-b19f-2db732145b53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:58:00.736156 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:00.736121    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:58:00.963157 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:00.963121    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a35f644c-4064-4099-8721-d796bddb0644-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-sdzrz\" (UID: \"a35f644c-4064-4099-8721-d796bddb0644\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sdzrz"
Sep 17 13:58:00.965802 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:00.965770    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a35f644c-4064-4099-8721-d796bddb0644-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-sdzrz\" (UID: \"a35f644c-4064-4099-8721-d796bddb0644\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sdzrz"
Sep 17 13:58:01.149339 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:01.149309    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sdzrz"
Sep 17 13:58:01.164944 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:01.164915    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a29762f9-e0d5-43cd-b19f-2db732145b53-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a29762f9-e0d5-43cd-b19f-2db732145b53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:58:01.167578 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:01.167553    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a29762f9-e0d5-43cd-b19f-2db732145b53-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a29762f9-e0d5-43cd-b19f-2db732145b53\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:58:01.440385 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:01.440354    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:58:02.527018 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:02.526989    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-754b4f6bdc-wngxz"]
Sep 17 13:58:02.531032 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:02.531000    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-754b4f6bdc-wngxz"
Sep 17 13:58:02.534535 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:02.534515    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 17 13:58:02.534651 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:02.534627    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-nqc5c\""
Sep 17 13:58:02.534835 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:02.534817    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 17 13:58:02.534924 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:02.534841    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 17 13:58:02.536250 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:02.536230    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 17 13:58:02.536337 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:02.536230    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 17 13:58:02.538635 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:02.538604    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-2akv2djqrcvk1\""
Sep 17 13:58:02.544713 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:02.544686    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:58:02.556610 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:02.556578    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-754b4f6bdc-wngxz"]
Sep 17 13:58:02.579227 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:02.579204    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6ee51de3-c49d-45aa-addb-41d96f782d1e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-754b4f6bdc-wngxz\" (UID: \"6ee51de3-c49d-45aa-addb-41d96f782d1e\") " pod="openshift-monitoring/thanos-querier-754b4f6bdc-wngxz"
Sep 17 13:58:02.579331 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:02.579254    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ee51de3-c49d-45aa-addb-41d96f782d1e-metrics-client-ca\") pod \"thanos-querier-754b4f6bdc-wngxz\" (UID: \"6ee51de3-c49d-45aa-addb-41d96f782d1e\") " pod="openshift-monitoring/thanos-querier-754b4f6bdc-wngxz"
Sep 17 13:58:02.579387 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:02.579338    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6ee51de3-c49d-45aa-addb-41d96f782d1e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-754b4f6bdc-wngxz\" (UID: \"6ee51de3-c49d-45aa-addb-41d96f782d1e\") " pod="openshift-monitoring/thanos-querier-754b4f6bdc-wngxz"
Sep 17 13:58:02.579387 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:02.579370    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6ee51de3-c49d-45aa-addb-41d96f782d1e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-754b4f6bdc-wngxz\" (UID: \"6ee51de3-c49d-45aa-addb-41d96f782d1e\") " pod="openshift-monitoring/thanos-querier-754b4f6bdc-wngxz"
Sep 17 13:58:02.579495 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:02.579395    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6ee51de3-c49d-45aa-addb-41d96f782d1e-secret-thanos-querier-tls\") pod \"thanos-querier-754b4f6bdc-wngxz\" (UID: \"6ee51de3-c49d-45aa-addb-41d96f782d1e\") " pod="openshift-monitoring/thanos-querier-754b4f6bdc-wngxz"
Sep 17 13:58:02.579495 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:02.579432    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6ee51de3-c49d-45aa-addb-41d96f782d1e-secret-grpc-tls\") pod \"thanos-querier-754b4f6bdc-wngxz\" (UID: \"6ee51de3-c49d-45aa-addb-41d96f782d1e\") " pod="openshift-monitoring/thanos-querier-754b4f6bdc-wngxz"
Sep 17 13:58:02.579582 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:02.579514    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh8ph\" (UniqueName: \"kubernetes.io/projected/6ee51de3-c49d-45aa-addb-41d96f782d1e-kube-api-access-kh8ph\") pod \"thanos-querier-754b4f6bdc-wngxz\" (UID: \"6ee51de3-c49d-45aa-addb-41d96f782d1e\") " pod="openshift-monitoring/thanos-querier-754b4f6bdc-wngxz"
Sep 17 13:58:02.579582 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:02.579537    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6ee51de3-c49d-45aa-addb-41d96f782d1e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-754b4f6bdc-wngxz\" (UID: \"6ee51de3-c49d-45aa-addb-41d96f782d1e\") " pod="openshift-monitoring/thanos-querier-754b4f6bdc-wngxz"
Sep 17 13:58:02.637168 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:02.637138    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:58:02.680862 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:02.680830    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6ee51de3-c49d-45aa-addb-41d96f782d1e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-754b4f6bdc-wngxz\" (UID: \"6ee51de3-c49d-45aa-addb-41d96f782d1e\") " pod="openshift-monitoring/thanos-querier-754b4f6bdc-wngxz"
Sep 17 13:58:02.681028 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:02.680876    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6ee51de3-c49d-45aa-addb-41d96f782d1e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-754b4f6bdc-wngxz\" (UID: \"6ee51de3-c49d-45aa-addb-41d96f782d1e\") " pod="openshift-monitoring/thanos-querier-754b4f6bdc-wngxz"
Sep 17 13:58:02.681028 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:02.680909    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6ee51de3-c49d-45aa-addb-41d96f782d1e-secret-thanos-querier-tls\") pod \"thanos-querier-754b4f6bdc-wngxz\" (UID: \"6ee51de3-c49d-45aa-addb-41d96f782d1e\") " pod="openshift-monitoring/thanos-querier-754b4f6bdc-wngxz"
Sep 17 13:58:02.681028 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:02.680938    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6ee51de3-c49d-45aa-addb-41d96f782d1e-secret-grpc-tls\") pod \"thanos-querier-754b4f6bdc-wngxz\" (UID: \"6ee51de3-c49d-45aa-addb-41d96f782d1e\") " pod="openshift-monitoring/thanos-querier-754b4f6bdc-wngxz"
Sep 17 13:58:02.681028 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:02.681002    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh8ph\" (UniqueName: \"kubernetes.io/projected/6ee51de3-c49d-45aa-addb-41d96f782d1e-kube-api-access-kh8ph\") pod \"thanos-querier-754b4f6bdc-wngxz\" (UID: \"6ee51de3-c49d-45aa-addb-41d96f782d1e\") " pod="openshift-monitoring/thanos-querier-754b4f6bdc-wngxz"
Sep 17 13:58:02.681241 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:02.681033    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6ee51de3-c49d-45aa-addb-41d96f782d1e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-754b4f6bdc-wngxz\" (UID: \"6ee51de3-c49d-45aa-addb-41d96f782d1e\") " pod="openshift-monitoring/thanos-querier-754b4f6bdc-wngxz"
Sep 17 13:58:02.681241 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:02.681099    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6ee51de3-c49d-45aa-addb-41d96f782d1e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-754b4f6bdc-wngxz\" (UID: \"6ee51de3-c49d-45aa-addb-41d96f782d1e\") " pod="openshift-monitoring/thanos-querier-754b4f6bdc-wngxz"
Sep 17 13:58:02.681241 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:02.681146    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ee51de3-c49d-45aa-addb-41d96f782d1e-metrics-client-ca\") pod \"thanos-querier-754b4f6bdc-wngxz\" (UID: \"6ee51de3-c49d-45aa-addb-41d96f782d1e\") " pod="openshift-monitoring/thanos-querier-754b4f6bdc-wngxz"
Sep 17 13:58:02.681999 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:02.681961    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ee51de3-c49d-45aa-addb-41d96f782d1e-metrics-client-ca\") pod \"thanos-querier-754b4f6bdc-wngxz\" (UID: \"6ee51de3-c49d-45aa-addb-41d96f782d1e\") " pod="openshift-monitoring/thanos-querier-754b4f6bdc-wngxz"
Sep 17 13:58:02.684435 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:02.684408    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6ee51de3-c49d-45aa-addb-41d96f782d1e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-754b4f6bdc-wngxz\" (UID: \"6ee51de3-c49d-45aa-addb-41d96f782d1e\") " pod="openshift-monitoring/thanos-querier-754b4f6bdc-wngxz"
Sep 17 13:58:02.684682 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:02.684657    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6ee51de3-c49d-45aa-addb-41d96f782d1e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-754b4f6bdc-wngxz\" (UID: \"6ee51de3-c49d-45aa-addb-41d96f782d1e\") " pod="openshift-monitoring/thanos-querier-754b4f6bdc-wngxz"
Sep 17 13:58:02.684920 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:02.684896    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6ee51de3-c49d-45aa-addb-41d96f782d1e-secret-grpc-tls\") pod \"thanos-querier-754b4f6bdc-wngxz\" (UID: \"6ee51de3-c49d-45aa-addb-41d96f782d1e\") " pod="openshift-monitoring/thanos-querier-754b4f6bdc-wngxz"
Sep 17 13:58:02.685137 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:02.685115    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6ee51de3-c49d-45aa-addb-41d96f782d1e-secret-thanos-querier-tls\") pod \"thanos-querier-754b4f6bdc-wngxz\" (UID: \"6ee51de3-c49d-45aa-addb-41d96f782d1e\") " pod="openshift-monitoring/thanos-querier-754b4f6bdc-wngxz"
Sep 17 13:58:02.685401 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:02.685376    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6ee51de3-c49d-45aa-addb-41d96f782d1e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-754b4f6bdc-wngxz\" (UID: \"6ee51de3-c49d-45aa-addb-41d96f782d1e\") " pod="openshift-monitoring/thanos-querier-754b4f6bdc-wngxz"
Sep 17 13:58:02.685495 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:02.685407    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6ee51de3-c49d-45aa-addb-41d96f782d1e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-754b4f6bdc-wngxz\" (UID: \"6ee51de3-c49d-45aa-addb-41d96f782d1e\") " pod="openshift-monitoring/thanos-querier-754b4f6bdc-wngxz"
Sep 17 13:58:02.699220 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:02.699199    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh8ph\" (UniqueName: \"kubernetes.io/projected/6ee51de3-c49d-45aa-addb-41d96f782d1e-kube-api-access-kh8ph\") pod \"thanos-querier-754b4f6bdc-wngxz\" (UID: \"6ee51de3-c49d-45aa-addb-41d96f782d1e\") " pod="openshift-monitoring/thanos-querier-754b4f6bdc-wngxz"
Sep 17 13:58:02.842650 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:02.842617    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-754b4f6bdc-wngxz"
Sep 17 13:58:04.039024 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:04.038986    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:58:04.134348 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:04.134306    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:58:04.169657 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:04.169619    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-gdznh"]
Sep 17 13:58:04.172919 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:04.172890    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-gdznh"
Sep 17 13:58:04.176924 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:04.176898    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 17 13:58:04.177075 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:04.176946    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-fsr6v\""
Sep 17 13:58:04.195902 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:04.195876    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6l5bf\" (UniqueName: \"kubernetes.io/projected/6b5113b4-136d-4d6c-a435-b0d9e7ea6232-kube-api-access-6l5bf\") pod \"monitoring-plugin-5d85778b9d-gdznh\" (UID: \"6b5113b4-136d-4d6c-a435-b0d9e7ea6232\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-gdznh"
Sep 17 13:58:04.196037 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:04.195928    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6b5113b4-136d-4d6c-a435-b0d9e7ea6232-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-gdznh\" (UID: \"6b5113b4-136d-4d6c-a435-b0d9e7ea6232\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-gdznh"
Sep 17 13:58:04.197472 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:04.197430    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-gdznh"]
Sep 17 13:58:04.296952 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:04.296866    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l5bf\" (UniqueName: \"kubernetes.io/projected/6b5113b4-136d-4d6c-a435-b0d9e7ea6232-kube-api-access-6l5bf\") pod \"monitoring-plugin-5d85778b9d-gdznh\" (UID: \"6b5113b4-136d-4d6c-a435-b0d9e7ea6232\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-gdznh"
Sep 17 13:58:04.296952 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:04.296909    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6b5113b4-136d-4d6c-a435-b0d9e7ea6232-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-gdznh\" (UID: \"6b5113b4-136d-4d6c-a435-b0d9e7ea6232\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-gdznh"
Sep 17 13:58:04.299624 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:04.299595    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6b5113b4-136d-4d6c-a435-b0d9e7ea6232-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-gdznh\" (UID: \"6b5113b4-136d-4d6c-a435-b0d9e7ea6232\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-gdznh"
Sep 17 13:58:04.306445 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:04.306423    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6l5bf\" (UniqueName: \"kubernetes.io/projected/6b5113b4-136d-4d6c-a435-b0d9e7ea6232-kube-api-access-6l5bf\") pod \"monitoring-plugin-5d85778b9d-gdznh\" (UID: \"6b5113b4-136d-4d6c-a435-b0d9e7ea6232\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-gdznh"
Sep 17 13:58:04.483997 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:04.483962    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-gdznh"
Sep 17 13:58:04.838608 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:04.838570    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:58:05.328505 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:05.328452    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-fb686c6f6-58glc"
Sep 17 13:58:05.328900 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:05.328534    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-fb686c6f6-58glc"
Sep 17 13:58:05.334875 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:05.334853    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-fb686c6f6-58glc"
Sep 17 13:58:05.890381 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:05.890326    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:58:06.164128 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:06.164037    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-fb686c6f6-58glc"
Sep 17 13:58:06.847873 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:06.847845    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-688858bc96-kqg68"]
Sep 17 13:58:06.854066 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:06.854042    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-688858bc96-kqg68"
Sep 17 13:58:06.863846 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:06.863822    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-688858bc96-kqg68"]
Sep 17 13:58:06.864344 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:06.864317    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 17 13:58:06.922934 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:06.922905    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46cd612c-7614-48d7-823f-115a955f2856-trusted-ca-bundle\") pod \"console-688858bc96-kqg68\" (UID: \"46cd612c-7614-48d7-823f-115a955f2856\") " pod="openshift-console/console-688858bc96-kqg68"
Sep 17 13:58:06.923081 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:06.922950    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/46cd612c-7614-48d7-823f-115a955f2856-console-config\") pod \"console-688858bc96-kqg68\" (UID: \"46cd612c-7614-48d7-823f-115a955f2856\") " pod="openshift-console/console-688858bc96-kqg68"
Sep 17 13:58:06.923081 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:06.923013    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/46cd612c-7614-48d7-823f-115a955f2856-console-oauth-config\") pod \"console-688858bc96-kqg68\" (UID: \"46cd612c-7614-48d7-823f-115a955f2856\") " pod="openshift-console/console-688858bc96-kqg68"
Sep 17 13:58:06.923081 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:06.923046    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/46cd612c-7614-48d7-823f-115a955f2856-console-serving-cert\") pod \"console-688858bc96-kqg68\" (UID: \"46cd612c-7614-48d7-823f-115a955f2856\") " pod="openshift-console/console-688858bc96-kqg68"
Sep 17 13:58:06.923236 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:06.923097    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/46cd612c-7614-48d7-823f-115a955f2856-service-ca\") pod \"console-688858bc96-kqg68\" (UID: \"46cd612c-7614-48d7-823f-115a955f2856\") " pod="openshift-console/console-688858bc96-kqg68"
Sep 17 13:58:06.923236 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:06.923141    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mszpq\" (UniqueName: \"kubernetes.io/projected/46cd612c-7614-48d7-823f-115a955f2856-kube-api-access-mszpq\") pod \"console-688858bc96-kqg68\" (UID: \"46cd612c-7614-48d7-823f-115a955f2856\") " pod="openshift-console/console-688858bc96-kqg68"
Sep 17 13:58:06.923332 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:06.923232    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/46cd612c-7614-48d7-823f-115a955f2856-oauth-serving-cert\") pod \"console-688858bc96-kqg68\" (UID: \"46cd612c-7614-48d7-823f-115a955f2856\") " pod="openshift-console/console-688858bc96-kqg68"
Sep 17 13:58:07.024148 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:07.024110    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46cd612c-7614-48d7-823f-115a955f2856-trusted-ca-bundle\") pod \"console-688858bc96-kqg68\" (UID: \"46cd612c-7614-48d7-823f-115a955f2856\") " pod="openshift-console/console-688858bc96-kqg68"
Sep 17 13:58:07.024316 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:07.024157    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/46cd612c-7614-48d7-823f-115a955f2856-console-config\") pod \"console-688858bc96-kqg68\" (UID: \"46cd612c-7614-48d7-823f-115a955f2856\") " pod="openshift-console/console-688858bc96-kqg68"
Sep 17 13:58:07.024316 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:07.024216    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/46cd612c-7614-48d7-823f-115a955f2856-console-oauth-config\") pod \"console-688858bc96-kqg68\" (UID: \"46cd612c-7614-48d7-823f-115a955f2856\") " pod="openshift-console/console-688858bc96-kqg68"
Sep 17 13:58:07.024316 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:07.024244    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/46cd612c-7614-48d7-823f-115a955f2856-console-serving-cert\") pod \"console-688858bc96-kqg68\" (UID: \"46cd612c-7614-48d7-823f-115a955f2856\") " pod="openshift-console/console-688858bc96-kqg68"
Sep 17 13:58:07.024316 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:07.024297    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/46cd612c-7614-48d7-823f-115a955f2856-service-ca\") pod \"console-688858bc96-kqg68\" (UID: \"46cd612c-7614-48d7-823f-115a955f2856\") " pod="openshift-console/console-688858bc96-kqg68"
Sep 17 13:58:07.024560 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:07.024349    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mszpq\" (UniqueName: \"kubernetes.io/projected/46cd612c-7614-48d7-823f-115a955f2856-kube-api-access-mszpq\") pod \"console-688858bc96-kqg68\" (UID: \"46cd612c-7614-48d7-823f-115a955f2856\") " pod="openshift-console/console-688858bc96-kqg68"
Sep 17 13:58:07.024560 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:07.024414    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/46cd612c-7614-48d7-823f-115a955f2856-oauth-serving-cert\") pod \"console-688858bc96-kqg68\" (UID: \"46cd612c-7614-48d7-823f-115a955f2856\") " pod="openshift-console/console-688858bc96-kqg68"
Sep 17 13:58:07.025062 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:07.025018    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/46cd612c-7614-48d7-823f-115a955f2856-console-config\") pod \"console-688858bc96-kqg68\" (UID: \"46cd612c-7614-48d7-823f-115a955f2856\") " pod="openshift-console/console-688858bc96-kqg68"
Sep 17 13:58:07.025166 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:07.025084    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/46cd612c-7614-48d7-823f-115a955f2856-oauth-serving-cert\") pod \"console-688858bc96-kqg68\" (UID: \"46cd612c-7614-48d7-823f-115a955f2856\") " pod="openshift-console/console-688858bc96-kqg68"
Sep 17 13:58:07.025166 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:07.025104    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/46cd612c-7614-48d7-823f-115a955f2856-service-ca\") pod \"console-688858bc96-kqg68\" (UID: \"46cd612c-7614-48d7-823f-115a955f2856\") " pod="openshift-console/console-688858bc96-kqg68"
Sep 17 13:58:07.025166 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:07.025105    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46cd612c-7614-48d7-823f-115a955f2856-trusted-ca-bundle\") pod \"console-688858bc96-kqg68\" (UID: \"46cd612c-7614-48d7-823f-115a955f2856\") " pod="openshift-console/console-688858bc96-kqg68"
Sep 17 13:58:07.027672 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:07.027647    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/46cd612c-7614-48d7-823f-115a955f2856-console-serving-cert\") pod \"console-688858bc96-kqg68\" (UID: \"46cd612c-7614-48d7-823f-115a955f2856\") " pod="openshift-console/console-688858bc96-kqg68"
Sep 17 13:58:07.027672 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:07.027655    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/46cd612c-7614-48d7-823f-115a955f2856-console-oauth-config\") pod \"console-688858bc96-kqg68\" (UID: \"46cd612c-7614-48d7-823f-115a955f2856\") " pod="openshift-console/console-688858bc96-kqg68"
Sep 17 13:58:07.034075 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:07.034056    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mszpq\" (UniqueName: \"kubernetes.io/projected/46cd612c-7614-48d7-823f-115a955f2856-kube-api-access-mszpq\") pod \"console-688858bc96-kqg68\" (UID: \"46cd612c-7614-48d7-823f-115a955f2856\") " pod="openshift-console/console-688858bc96-kqg68"
Sep 17 13:58:07.165615 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:07.165545    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-688858bc96-kqg68"
Sep 17 13:58:07.296083 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:58:07.295298    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ae12eb6_f825_4313_8884_1e879544e4f6.slice/crio-b34164cadce6ceb2b0953e8ad5bc716123ad64fd09d9fb063292857dfe0d01b7 WatchSource:0}: Error finding container b34164cadce6ceb2b0953e8ad5bc716123ad64fd09d9fb063292857dfe0d01b7: Status 404 returned error can't find the container with id b34164cadce6ceb2b0953e8ad5bc716123ad64fd09d9fb063292857dfe0d01b7
Sep 17 13:58:07.360405 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:07.360012    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:58:07.452530 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:07.452500    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:58:07.575941 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:07.575752    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-754b4f6bdc-wngxz"]
Sep 17 13:58:07.577638 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:58:07.577607    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ee51de3_c49d_45aa_addb_41d96f782d1e.slice/crio-dafbf013f6bf60090a5e1cb3425048bbe15fd2ae893c90842113f0c176bbffc8 WatchSource:0}: Error finding container dafbf013f6bf60090a5e1cb3425048bbe15fd2ae893c90842113f0c176bbffc8: Status 404 returned error can't find the container with id dafbf013f6bf60090a5e1cb3425048bbe15fd2ae893c90842113f0c176bbffc8
Sep 17 13:58:07.590579 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:07.590553    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-gdznh"]
Sep 17 13:58:07.594163 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:07.594144    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-blrzn"]
Sep 17 13:58:07.594420 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:58:07.594399    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b5113b4_136d_4d6c_a435_b0d9e7ea6232.slice/crio-c6c023994970a87b8b8cb13146a48dca0684fa997d5eb0147595e2d795969329 WatchSource:0}: Error finding container c6c023994970a87b8b8cb13146a48dca0684fa997d5eb0147595e2d795969329: Status 404 returned error can't find the container with id c6c023994970a87b8b8cb13146a48dca0684fa997d5eb0147595e2d795969329
Sep 17 13:58:07.597884 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:07.597856    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-sdzrz"]
Sep 17 13:58:07.599027 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:07.598924    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:58:07.599518 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:58:07.599497    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb89d8a9d_a0d5_4578_bf8a_1711c9e5646a.slice/crio-526c16e1988bf6c0fe284644f003608fae16a4d1c93c5add3acd7d39e6f06bff WatchSource:0}: Error finding container 526c16e1988bf6c0fe284644f003608fae16a4d1c93c5add3acd7d39e6f06bff: Status 404 returned error can't find the container with id 526c16e1988bf6c0fe284644f003608fae16a4d1c93c5add3acd7d39e6f06bff
Sep 17 13:58:07.627286 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:07.627265    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-688858bc96-kqg68"]
Sep 17 13:58:07.630657 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:07.630639    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 13:58:07.635772 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:58:07.635735    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46cd612c_7614_48d7_823f_115a955f2856.slice/crio-7c01531bc2320e8efafe2c1de29ee0ca224bf525a1c81f39f6c0d590cc72f882 WatchSource:0}: Error finding container 7c01531bc2320e8efafe2c1de29ee0ca224bf525a1c81f39f6c0d590cc72f882: Status 404 returned error can't find the container with id 7c01531bc2320e8efafe2c1de29ee0ca224bf525a1c81f39f6c0d590cc72f882
Sep 17 13:58:07.636107 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:58:07.636085    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda29762f9_e0d5_43cd_b19f_2db732145b53.slice/crio-3bfcdc3744fe01b27fc183c70fec18087250212c11ccc0bd1d1a8f685a77f98f WatchSource:0}: Error finding container 3bfcdc3744fe01b27fc183c70fec18087250212c11ccc0bd1d1a8f685a77f98f: Status 404 returned error can't find the container with id 3bfcdc3744fe01b27fc183c70fec18087250212c11ccc0bd1d1a8f685a77f98f
Sep 17 13:58:07.698139 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:07.697992    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:58:08.167384 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:08.167342    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sdzrz" event={"ID":"a35f644c-4064-4099-8721-d796bddb0644","Type":"ContainerStarted","Data":"45b9dcc5df02b5048858d7cdfd7da5a08d5f18009388f3eb73acab46ddcf7851"}
Sep 17 13:58:08.168841 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:08.168805    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-gdznh" event={"ID":"6b5113b4-136d-4d6c-a435-b0d9e7ea6232","Type":"ContainerStarted","Data":"c6c023994970a87b8b8cb13146a48dca0684fa997d5eb0147595e2d795969329"}
Sep 17 13:58:08.170248 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:08.170219    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gqkh8" event={"ID":"7ae12eb6-f825-4313-8884-1e879544e4f6","Type":"ContainerStarted","Data":"b34164cadce6ceb2b0953e8ad5bc716123ad64fd09d9fb063292857dfe0d01b7"}
Sep 17 13:58:08.171593 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:08.171566    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-754b4f6bdc-wngxz" event={"ID":"6ee51de3-c49d-45aa-addb-41d96f782d1e","Type":"ContainerStarted","Data":"dafbf013f6bf60090a5e1cb3425048bbe15fd2ae893c90842113f0c176bbffc8"}
Sep 17 13:58:08.173422 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:08.173394    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-kpx6x" event={"ID":"a2d73565-b83f-4582-9c80-84c460828fe6","Type":"ContainerStarted","Data":"89006a0627ed53a5b3028af7e40d1a2e4e9454f042a05426541d83c0cc651f4c"}
Sep 17 13:58:08.173672 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:08.173644    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-kpx6x"
Sep 17 13:58:08.175686 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:08.175416    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-688858bc96-kqg68" event={"ID":"46cd612c-7614-48d7-823f-115a955f2856","Type":"ContainerStarted","Data":"8fbfb6375f3a19264dc26598ef2c4204e0130387203e48c3f8014673b0bf57d8"}
Sep 17 13:58:08.175857 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:08.175456    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-688858bc96-kqg68" event={"ID":"46cd612c-7614-48d7-823f-115a955f2856","Type":"ContainerStarted","Data":"7c01531bc2320e8efafe2c1de29ee0ca224bf525a1c81f39f6c0d590cc72f882"}
Sep 17 13:58:08.178288 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:08.178259    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-blrzn" event={"ID":"b89d8a9d-a0d5-4578-bf8a-1711c9e5646a","Type":"ContainerStarted","Data":"569d4784f65913810daff5227b51f7054ae905ce59961e1035a3d1aa2ff99c73"}
Sep 17 13:58:08.178392 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:08.178320    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-blrzn" event={"ID":"b89d8a9d-a0d5-4578-bf8a-1711c9e5646a","Type":"ContainerStarted","Data":"0d56199098f77833b4d75304d6604f6f850c08d0633b1a35c4bcce4bd4ef4613"}
Sep 17 13:58:08.178392 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:08.178355    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-blrzn" event={"ID":"b89d8a9d-a0d5-4578-bf8a-1711c9e5646a","Type":"ContainerStarted","Data":"526c16e1988bf6c0fe284644f003608fae16a4d1c93c5add3acd7d39e6f06bff"}
Sep 17 13:58:08.180273 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:08.180254    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a29762f9-e0d5-43cd-b19f-2db732145b53","Type":"ContainerStarted","Data":"3bfcdc3744fe01b27fc183c70fec18087250212c11ccc0bd1d1a8f685a77f98f"}
Sep 17 13:58:08.184076 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:08.184035    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-kpx6x"
Sep 17 13:58:08.194724 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:08.194623    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-kpx6x" podStartSLOduration=1.752835697 podStartE2EDuration="18.194605769s" totalImagesPullingTime="16.441770072s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 13:57:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 13:58:08.193417728 +0000 UTC m=+157.135561468" watchObservedRunningTime="2026-09-17 13:58:08.194605769 +0000 UTC m=+157.136749415"
Sep 17 13:58:09.399849 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:58:09.399803    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-d47ffd9b-tr5bs" podUID="fb692cc1-e158-4666-baca-0d719d8c8aa6"
Sep 17 13:58:09.416930 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:58:09.416874    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-82dhq" podUID="45035743-85eb-4642-a5c9-adea5c5db4e1"
Sep 17 13:58:10.195277 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:10.195219    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-82dhq"
Sep 17 13:58:10.195736 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:10.195714    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d47ffd9b-tr5bs"
Sep 17 13:58:10.619764 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:10.619729    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:58:10.696794 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:10.696752    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:58:10.872248 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:10.872155    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:58:10.950338 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:10.950304    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:58:11.201274 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:11.201190    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gqkh8" event={"ID":"7ae12eb6-f825-4313-8884-1e879544e4f6","Type":"ContainerStarted","Data":"1c05bf87649e13e8f90d8b32ca4dc3ac7ad43e0eb6e040f48440e602a457cfb0"}
Sep 17 13:58:11.227156 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:11.227101    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-688858bc96-kqg68" podStartSLOduration=5.227080583 podStartE2EDuration="5.227080583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 13:58:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 13:58:08.235625043 +0000 UTC m=+157.177768691" watchObservedRunningTime="2026-09-17 13:58:11.227080583 +0000 UTC m=+160.169224226"
Sep 17 13:58:12.207480 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:12.207434    2560 generic.go:356] "Generic (PLEG): container finished" podID="7ae12eb6-f825-4313-8884-1e879544e4f6" containerID="1c05bf87649e13e8f90d8b32ca4dc3ac7ad43e0eb6e040f48440e602a457cfb0" exitCode=0
Sep 17 13:58:12.208008 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:12.207503    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gqkh8" event={"ID":"7ae12eb6-f825-4313-8884-1e879544e4f6","Type":"ContainerDied","Data":"1c05bf87649e13e8f90d8b32ca4dc3ac7ad43e0eb6e040f48440e602a457cfb0"}
Sep 17 13:58:12.710614 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:12.710309    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-d47ffd9b-tr5bs"]
Sep 17 13:58:12.710747 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:58:12.710612    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-d47ffd9b-tr5bs" podUID="fb692cc1-e158-4666-baca-0d719d8c8aa6"
Sep 17 13:58:12.846784 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:12.846722    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:58:12.951425 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:12.951398    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:58:13.214387 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:13.214334    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gqkh8" event={"ID":"7ae12eb6-f825-4313-8884-1e879544e4f6","Type":"ContainerStarted","Data":"bb7e680fcc1b488d13ebe85b63c7e7c103b40d890b020a70f0deb49302d8e34e"}
Sep 17 13:58:13.214838 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:13.214427    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gqkh8" event={"ID":"7ae12eb6-f825-4313-8884-1e879544e4f6","Type":"ContainerStarted","Data":"991dea40cb039f30850bdf473a4d40581f5f48051e10a5e75ac951a154a4e3ed"}
Sep 17 13:58:13.217057 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:13.217029    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-754b4f6bdc-wngxz" event={"ID":"6ee51de3-c49d-45aa-addb-41d96f782d1e","Type":"ContainerStarted","Data":"8498703c03bb9fbc5de3a953ffbb3bbf1d7c163923b18bcda244042871b601b1"}
Sep 17 13:58:13.217192 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:13.217100    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-754b4f6bdc-wngxz" event={"ID":"6ee51de3-c49d-45aa-addb-41d96f782d1e","Type":"ContainerStarted","Data":"c6e4caa735c53db27b54ffb0529e5f33470f71478f4deb7133ff8d6af489ac19"}
Sep 17 13:58:13.217192 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:13.217144    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-754b4f6bdc-wngxz" event={"ID":"6ee51de3-c49d-45aa-addb-41d96f782d1e","Type":"ContainerStarted","Data":"d18f0c8dfdfe0cb2b3ab9ad2584aa4d16cab1064fe00afdea43f6a9acd826cd9"}
Sep 17 13:58:13.219489 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:13.219447    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-blrzn" event={"ID":"b89d8a9d-a0d5-4578-bf8a-1711c9e5646a","Type":"ContainerStarted","Data":"4fe6522ec1e028b265f0147ae2e2f55259eabd2606f920bde2c9208b27885297"}
Sep 17 13:58:13.221334 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:13.221307    2560 generic.go:356] "Generic (PLEG): container finished" podID="a29762f9-e0d5-43cd-b19f-2db732145b53" containerID="f43b8b6cb2c544eb1cf4de34c863c18f03b94303fac98bf70955d30062fe1f51" exitCode=0
Sep 17 13:58:13.221454 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:13.221390    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a29762f9-e0d5-43cd-b19f-2db732145b53","Type":"ContainerDied","Data":"f43b8b6cb2c544eb1cf4de34c863c18f03b94303fac98bf70955d30062fe1f51"}
Sep 17 13:58:13.223981 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:13.223958    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sdzrz" event={"ID":"a35f644c-4064-4099-8721-d796bddb0644","Type":"ContainerStarted","Data":"bf11ac6258ac19eb2c636533dbfae716d11ac81ff870e0d018c376c5f4a9c083"}
Sep 17 13:58:13.224092 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:13.224014    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sdzrz" event={"ID":"a35f644c-4064-4099-8721-d796bddb0644","Type":"ContainerStarted","Data":"72a1df0a71825ff95a16058859096447f4b94b6bc0e25c0d1aeccf42ca7f8bf4"}
Sep 17 13:58:13.224092 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:13.224044    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sdzrz" event={"ID":"a35f644c-4064-4099-8721-d796bddb0644","Type":"ContainerStarted","Data":"ded2b947aa6db38c77bfa6e0cd9c89f54439a525cc68b654c301c72025f1ae3a"}
Sep 17 13:58:13.226048 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:13.226025    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-gdznh" event={"ID":"6b5113b4-136d-4d6c-a435-b0d9e7ea6232","Type":"ContainerStarted","Data":"4843a8abd111e0c6fcd445e80680947c14d969bbeafa953e7a99390df4aded29"}
Sep 17 13:58:13.226151 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:13.226068    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d47ffd9b-tr5bs"
Sep 17 13:58:13.226299 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:13.226282    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-gdznh"
Sep 17 13:58:13.231944 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:13.231880    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-gdznh"
Sep 17 13:58:13.248412 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:13.248387    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d47ffd9b-tr5bs"
Sep 17 13:58:13.278685 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:13.278630    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-gdznh" podStartSLOduration=4.523352463 podStartE2EDuration="9.27861259s" totalImagesPullingTime="4.755260127s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 13:58:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 13:58:13.278108348 +0000 UTC m=+162.220251994" watchObservedRunningTime="2026-09-17 13:58:13.27861259 +0000 UTC m=+162.220756237"
Sep 17 13:58:13.279324 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:13.279290    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-gqkh8" podStartSLOduration=12.553824607 podStartE2EDuration="14.279281251s" totalImagesPullingTime="1.632950296s" totalInitContainerRuntime="92.506348ms" isStatefulPod=true podCreationTimestamp="2026-09-17 13:57:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 13:58:13.243952532 +0000 UTC m=+162.186096175" watchObservedRunningTime="2026-09-17 13:58:13.279281251 +0000 UTC m=+162.221424894"
Sep 17 13:58:13.321208 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:13.321156    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-blrzn" podStartSLOduration=9.706800743 podStartE2EDuration="14.321137338s" totalImagesPullingTime="4.614336595s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 13:57:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 13:58:13.320170098 +0000 UTC m=+162.262313754" watchObservedRunningTime="2026-09-17 13:58:13.321137338 +0000 UTC m=+162.263280985"
Sep 17 13:58:13.389361 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:13.389303    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sdzrz" podStartSLOduration=9.641825295 podStartE2EDuration="14.389284293s" totalImagesPullingTime="4.747458998s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 13:57:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 13:58:13.38919093 +0000 UTC m=+162.331334580" watchObservedRunningTime="2026-09-17 13:58:13.389284293 +0000 UTC m=+162.331427941"
Sep 17 13:58:13.396021 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:13.395996    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fb692cc1-e158-4666-baca-0d719d8c8aa6-image-registry-private-configuration\") pod \"fb692cc1-e158-4666-baca-0d719d8c8aa6\" (UID: \"fb692cc1-e158-4666-baca-0d719d8c8aa6\") "
Sep 17 13:58:13.396153 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:13.396062    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fb692cc1-e158-4666-baca-0d719d8c8aa6-bound-sa-token\") pod \"fb692cc1-e158-4666-baca-0d719d8c8aa6\" (UID: \"fb692cc1-e158-4666-baca-0d719d8c8aa6\") "
Sep 17 13:58:13.396210 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:13.396161    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2hhhr\" (UniqueName: \"kubernetes.io/projected/fb692cc1-e158-4666-baca-0d719d8c8aa6-kube-api-access-2hhhr\") pod \"fb692cc1-e158-4666-baca-0d719d8c8aa6\" (UID: \"fb692cc1-e158-4666-baca-0d719d8c8aa6\") "
Sep 17 13:58:13.396210 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:13.396190    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fb692cc1-e158-4666-baca-0d719d8c8aa6-registry-certificates\") pod \"fb692cc1-e158-4666-baca-0d719d8c8aa6\" (UID: \"fb692cc1-e158-4666-baca-0d719d8c8aa6\") "
Sep 17 13:58:13.396318 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:13.396225    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fb692cc1-e158-4666-baca-0d719d8c8aa6-installation-pull-secrets\") pod \"fb692cc1-e158-4666-baca-0d719d8c8aa6\" (UID: \"fb692cc1-e158-4666-baca-0d719d8c8aa6\") "
Sep 17 13:58:13.396318 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:13.396248    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fb692cc1-e158-4666-baca-0d719d8c8aa6-trusted-ca\") pod \"fb692cc1-e158-4666-baca-0d719d8c8aa6\" (UID: \"fb692cc1-e158-4666-baca-0d719d8c8aa6\") "
Sep 17 13:58:13.396318 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:13.396289    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fb692cc1-e158-4666-baca-0d719d8c8aa6-ca-trust-extracted\") pod \"fb692cc1-e158-4666-baca-0d719d8c8aa6\" (UID: \"fb692cc1-e158-4666-baca-0d719d8c8aa6\") "
Sep 17 13:58:13.396621 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:13.396591    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb692cc1-e158-4666-baca-0d719d8c8aa6-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "fb692cc1-e158-4666-baca-0d719d8c8aa6" (UID: "fb692cc1-e158-4666-baca-0d719d8c8aa6"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 13:58:13.396754 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:13.396723    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb692cc1-e158-4666-baca-0d719d8c8aa6-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "fb692cc1-e158-4666-baca-0d719d8c8aa6" (UID: "fb692cc1-e158-4666-baca-0d719d8c8aa6"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 13:58:13.396824 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:13.396727    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb692cc1-e158-4666-baca-0d719d8c8aa6-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "fb692cc1-e158-4666-baca-0d719d8c8aa6" (UID: "fb692cc1-e158-4666-baca-0d719d8c8aa6"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 13:58:13.398742 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:13.398713    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb692cc1-e158-4666-baca-0d719d8c8aa6-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "fb692cc1-e158-4666-baca-0d719d8c8aa6" (UID: "fb692cc1-e158-4666-baca-0d719d8c8aa6"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 13:58:13.398851 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:13.398822    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb692cc1-e158-4666-baca-0d719d8c8aa6-kube-api-access-2hhhr" (OuterVolumeSpecName: "kube-api-access-2hhhr") pod "fb692cc1-e158-4666-baca-0d719d8c8aa6" (UID: "fb692cc1-e158-4666-baca-0d719d8c8aa6"). InnerVolumeSpecName "kube-api-access-2hhhr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 13:58:13.398953 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:13.398923    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb692cc1-e158-4666-baca-0d719d8c8aa6-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "fb692cc1-e158-4666-baca-0d719d8c8aa6" (UID: "fb692cc1-e158-4666-baca-0d719d8c8aa6"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 13:58:13.398999 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:13.398978    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb692cc1-e158-4666-baca-0d719d8c8aa6-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "fb692cc1-e158-4666-baca-0d719d8c8aa6" (UID: "fb692cc1-e158-4666-baca-0d719d8c8aa6"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 13:58:13.497080 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:13.497017    2560 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fb692cc1-e158-4666-baca-0d719d8c8aa6-bound-sa-token\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 13:58:13.497080 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:13.497049    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hhhr\" (UniqueName: \"kubernetes.io/projected/fb692cc1-e158-4666-baca-0d719d8c8aa6-kube-api-access-2hhhr\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 13:58:13.497080 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:13.497063    2560 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fb692cc1-e158-4666-baca-0d719d8c8aa6-registry-certificates\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 13:58:13.497080 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:13.497077    2560 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fb692cc1-e158-4666-baca-0d719d8c8aa6-installation-pull-secrets\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 13:58:13.497318 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:13.497091    2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fb692cc1-e158-4666-baca-0d719d8c8aa6-trusted-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 13:58:13.497318 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:13.497105    2560 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fb692cc1-e158-4666-baca-0d719d8c8aa6-ca-trust-extracted\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 13:58:13.497318 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:13.497121    2560 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fb692cc1-e158-4666-baca-0d719d8c8aa6-image-registry-private-configuration\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 13:58:14.233210 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:14.233178    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d47ffd9b-tr5bs"
Sep 17 13:58:14.294875 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:14.294844    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-d47ffd9b-tr5bs"]
Sep 17 13:58:14.298328 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:14.298301    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-d47ffd9b-tr5bs"]
Sep 17 13:58:14.305477 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:14.305428    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fb692cc1-e158-4666-baca-0d719d8c8aa6-registry-tls\") pod \"image-registry-d47ffd9b-tr5bs\" (UID: \"fb692cc1-e158-4666-baca-0d719d8c8aa6\") " pod="openshift-image-registry/image-registry-d47ffd9b-tr5bs"
Sep 17 13:58:14.305602 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:14.305548    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/45035743-85eb-4642-a5c9-adea5c5db4e1-metrics-tls\") pod \"dns-default-82dhq\" (UID: \"45035743-85eb-4642-a5c9-adea5c5db4e1\") " pod="openshift-dns/dns-default-82dhq"
Sep 17 13:58:14.306315 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:58:14.306033    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: object "openshift-image-registry"/"image-registry-tls" not registered
Sep 17 13:58:14.306315 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:58:14.306056    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d47ffd9b-tr5bs: object "openshift-image-registry"/"image-registry-tls" not registered
Sep 17 13:58:14.306315 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:58:14.306111    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb692cc1-e158-4666-baca-0d719d8c8aa6-registry-tls podName:fb692cc1-e158-4666-baca-0d719d8c8aa6 nodeName:}" failed. No retries permitted until 2026-09-17 14:00:16.306090553 +0000 UTC m=+285.248234178 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fb692cc1-e158-4666-baca-0d719d8c8aa6-registry-tls") pod "image-registry-d47ffd9b-tr5bs" (UID: "fb692cc1-e158-4666-baca-0d719d8c8aa6") : object "openshift-image-registry"/"image-registry-tls" not registered
Sep 17 13:58:14.309119 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:14.309066    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/45035743-85eb-4642-a5c9-adea5c5db4e1-metrics-tls\") pod \"dns-default-82dhq\" (UID: \"45035743-85eb-4642-a5c9-adea5c5db4e1\") " pod="openshift-dns/dns-default-82dhq"
Sep 17 13:58:14.399760 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:14.399547    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-f8wrh\""
Sep 17 13:58:14.406651 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:14.406623    2560 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fb692cc1-e158-4666-baca-0d719d8c8aa6-registry-tls\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 13:58:14.407813 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:14.407786    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-82dhq"
Sep 17 13:58:14.872758 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:14.872725    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-82dhq"]
Sep 17 13:58:14.876210 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:58:14.876166    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45035743_85eb_4642_a5c9_adea5c5db4e1.slice/crio-240ed72602195cbebd7b41288e4a96c0514a1a4602f2aeda92e752eddd5220bc WatchSource:0}: Error finding container 240ed72602195cbebd7b41288e4a96c0514a1a4602f2aeda92e752eddd5220bc: Status 404 returned error can't find the container with id 240ed72602195cbebd7b41288e4a96c0514a1a4602f2aeda92e752eddd5220bc
Sep 17 13:58:15.240270 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:15.240139    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-82dhq" event={"ID":"45035743-85eb-4642-a5c9-adea5c5db4e1","Type":"ContainerStarted","Data":"240ed72602195cbebd7b41288e4a96c0514a1a4602f2aeda92e752eddd5220bc"}
Sep 17 13:58:15.245266 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:15.245233    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-754b4f6bdc-wngxz" event={"ID":"6ee51de3-c49d-45aa-addb-41d96f782d1e","Type":"ContainerStarted","Data":"023f53b180bd881259d9f552e82022748d07af8835073f7222f329067804cc5e"}
Sep 17 13:58:15.245401 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:15.245303    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-754b4f6bdc-wngxz" event={"ID":"6ee51de3-c49d-45aa-addb-41d96f782d1e","Type":"ContainerStarted","Data":"d3f47b7e8e420496b6799728aef98134de100ab17df2990db3e0482875bcfdb4"}
Sep 17 13:58:15.245401 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:15.245346    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-754b4f6bdc-wngxz" event={"ID":"6ee51de3-c49d-45aa-addb-41d96f782d1e","Type":"ContainerStarted","Data":"0a85a89f269d1eda948f5aa33b600bbe13d215ccba4221090f7178255de2100e"}
Sep 17 13:58:15.277436 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:15.277382    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-754b4f6bdc-wngxz" podStartSLOduration=6.705548567 podStartE2EDuration="13.277364772s" totalImagesPullingTime="6.571816205s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 13:58:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 13:58:15.275082847 +0000 UTC m=+164.217226493" watchObservedRunningTime="2026-09-17 13:58:15.277364772 +0000 UTC m=+164.219508417"
Sep 17 13:58:15.576907 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:15.576868    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb692cc1-e158-4666-baca-0d719d8c8aa6" path="/var/lib/kubelet/pods/fb692cc1-e158-4666-baca-0d719d8c8aa6/volumes"
Sep 17 13:58:16.243474 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:16.243437    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:58:16.249912 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:16.249884    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-754b4f6bdc-wngxz"
Sep 17 13:58:16.332694 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:16.332653    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:58:17.166730 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:17.166706    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-688858bc96-kqg68"
Sep 17 13:58:17.166811 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:17.166764    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-688858bc96-kqg68"
Sep 17 13:58:17.171722 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:17.171705    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-688858bc96-kqg68"
Sep 17 13:58:17.254514 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:17.254486    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a29762f9-e0d5-43cd-b19f-2db732145b53","Type":"ContainerStarted","Data":"5b21a340939172317b9e25530c46e14fb1f6e3c1bf3badb193c6a2257a230fb6"}
Sep 17 13:58:17.258678 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:17.258567    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-688858bc96-kqg68"
Sep 17 13:58:17.260723 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:17.260710    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-754b4f6bdc-wngxz"
Sep 17 13:58:17.326860 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:17.326832    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-fb686c6f6-58glc"]
Sep 17 13:58:17.459300 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:17.459221    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:58:17.658573 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:17.658544    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:58:17.759510 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:17.759480    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:58:18.260622 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:18.260586    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-82dhq" event={"ID":"45035743-85eb-4642-a5c9-adea5c5db4e1","Type":"ContainerStarted","Data":"f6d8f4ea33555720ee934dce5e92c399386ed6bafb4f4baf0667e16e0f1933a0"}
Sep 17 13:58:18.261171 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:18.260643    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-82dhq" event={"ID":"45035743-85eb-4642-a5c9-adea5c5db4e1","Type":"ContainerStarted","Data":"b12f33b7c5cc8d6be59359dc8431383898258b8058b0e371686b4c22516181ae"}
Sep 17 13:58:18.261171 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:18.260671    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-82dhq"
Sep 17 13:58:18.264011 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:18.263986    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a29762f9-e0d5-43cd-b19f-2db732145b53","Type":"ContainerStarted","Data":"b6b6e03dc4d2cc5227a5d40f01b0fb59984f1b058483c35d37b41869d5e846ab"}
Sep 17 13:58:18.264116 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:18.264041    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a29762f9-e0d5-43cd-b19f-2db732145b53","Type":"ContainerStarted","Data":"c67e8be4e049bd64c8436bb921e295fc72ce7c6542b8d55aa8b0ec847bf36ce8"}
Sep 17 13:58:18.264116 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:18.264072    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a29762f9-e0d5-43cd-b19f-2db732145b53","Type":"ContainerStarted","Data":"681dd32d424c8518284f25f74946ea91eccd7a3c8b8f125de8ffeb7f6a7149cb"}
Sep 17 13:58:18.264116 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:18.264100    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a29762f9-e0d5-43cd-b19f-2db732145b53","Type":"ContainerStarted","Data":"3fcaa8655428ec4e62d9404b4ec5ac72555b90b883987f9dd9b1e64910aa7e3f"}
Sep 17 13:58:18.264274 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:18.264129    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a29762f9-e0d5-43cd-b19f-2db732145b53","Type":"ContainerStarted","Data":"70d6616d4f35e34922b8c01a6195f2e3e7b167223a4a75ca01ad18f46b4c7734"}
Sep 17 13:58:18.281403 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:18.281363    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-82dhq" podStartSLOduration=129.690171042 podStartE2EDuration="2m12.2813489s" totalImagesPullingTime="2.591177858s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 13:56:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 13:58:18.279951602 +0000 UTC m=+167.222095252" watchObservedRunningTime="2026-09-17 13:58:18.2813489 +0000 UTC m=+167.223492546"
Sep 17 13:58:18.311120 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:18.311075    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=10.247561658 podStartE2EDuration="18.311060133s" totalImagesPullingTime="8.033703183s" totalInitContainerRuntime="29.795292ms" isStatefulPod=false podCreationTimestamp="2026-09-17 13:58:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 13:58:18.30911707 +0000 UTC m=+167.251260715" watchObservedRunningTime="2026-09-17 13:58:18.311060133 +0000 UTC m=+167.253203813"
Sep 17 13:58:25.612905 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:25.612808    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:58:25.684027 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:25.683994    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:58:28.271534 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:28.271502    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-82dhq"
Sep 17 13:58:31.821289 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:31.821260    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-82dhq_45035743-85eb-4642-a5c9-adea5c5db4e1/dns/0.log"
Sep 17 13:58:32.022068 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:32.022036    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-82dhq_45035743-85eb-4642-a5c9-adea5c5db4e1/kube-rbac-proxy/0.log"
Sep 17 13:58:32.622097 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:32.622072    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2zqvg_a4a8010a-d12b-4b1a-a0db-e1c64109b151/dns-node-resolver/0.log"
Sep 17 13:58:33.223372 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:33.223338    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-997fdcdb-82rfx_644fc778-b087-473e-91bf-44f9a89c2fef/router/0.log"
Sep 17 13:58:33.582363 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:33.582334    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:58:33.621292 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:33.621255    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wwqnd_2769cbbe-3a80-48c0-bf6b-a993360d1ca7/serve-healthcheck-canary/0.log"
Sep 17 13:58:33.697770 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:33.697734    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:58:33.761221 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:33.761190    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:58:34.213638 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:34.213606    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:58:34.274599 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:34.274575    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:58:37.237392 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:37.237351    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:58:37.304742 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:37.304711    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:58:42.346990 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:42.346940    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-fb686c6f6-58glc" podUID="878f435a-160a-4634-9ed2-e91c488b6854" containerName="console" containerID="cri-o://3d4032128f4155eb92089043d30cd830f0b15be0400551ba2125c133fc52d41a" gracePeriod=15
Sep 17 13:58:42.628899 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:42.628873    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fb686c6f6-58glc"
Sep 17 13:58:42.734694 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:42.734662    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/878f435a-160a-4634-9ed2-e91c488b6854-console-serving-cert\") pod \"878f435a-160a-4634-9ed2-e91c488b6854\" (UID: \"878f435a-160a-4634-9ed2-e91c488b6854\") "
Sep 17 13:58:42.734694 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:42.734698    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/878f435a-160a-4634-9ed2-e91c488b6854-service-ca\") pod \"878f435a-160a-4634-9ed2-e91c488b6854\" (UID: \"878f435a-160a-4634-9ed2-e91c488b6854\") "
Sep 17 13:58:42.734920 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:42.734720    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jhdt5\" (UniqueName: \"kubernetes.io/projected/878f435a-160a-4634-9ed2-e91c488b6854-kube-api-access-jhdt5\") pod \"878f435a-160a-4634-9ed2-e91c488b6854\" (UID: \"878f435a-160a-4634-9ed2-e91c488b6854\") "
Sep 17 13:58:42.734920 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:42.734738    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/878f435a-160a-4634-9ed2-e91c488b6854-console-config\") pod \"878f435a-160a-4634-9ed2-e91c488b6854\" (UID: \"878f435a-160a-4634-9ed2-e91c488b6854\") "
Sep 17 13:58:42.734920 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:42.734803    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/878f435a-160a-4634-9ed2-e91c488b6854-oauth-serving-cert\") pod \"878f435a-160a-4634-9ed2-e91c488b6854\" (UID: \"878f435a-160a-4634-9ed2-e91c488b6854\") "
Sep 17 13:58:42.734920 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:42.734866    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/878f435a-160a-4634-9ed2-e91c488b6854-console-oauth-config\") pod \"878f435a-160a-4634-9ed2-e91c488b6854\" (UID: \"878f435a-160a-4634-9ed2-e91c488b6854\") "
Sep 17 13:58:42.735230 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:42.735165    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/878f435a-160a-4634-9ed2-e91c488b6854-service-ca" (OuterVolumeSpecName: "service-ca") pod "878f435a-160a-4634-9ed2-e91c488b6854" (UID: "878f435a-160a-4634-9ed2-e91c488b6854"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 13:58:42.735230 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:42.735172    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/878f435a-160a-4634-9ed2-e91c488b6854-console-config" (OuterVolumeSpecName: "console-config") pod "878f435a-160a-4634-9ed2-e91c488b6854" (UID: "878f435a-160a-4634-9ed2-e91c488b6854"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 13:58:42.735444 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:42.735398    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/878f435a-160a-4634-9ed2-e91c488b6854-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "878f435a-160a-4634-9ed2-e91c488b6854" (UID: "878f435a-160a-4634-9ed2-e91c488b6854"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 13:58:42.737023 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:42.736997    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/878f435a-160a-4634-9ed2-e91c488b6854-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "878f435a-160a-4634-9ed2-e91c488b6854" (UID: "878f435a-160a-4634-9ed2-e91c488b6854"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 13:58:42.737104 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:42.737024    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/878f435a-160a-4634-9ed2-e91c488b6854-kube-api-access-jhdt5" (OuterVolumeSpecName: "kube-api-access-jhdt5") pod "878f435a-160a-4634-9ed2-e91c488b6854" (UID: "878f435a-160a-4634-9ed2-e91c488b6854"). InnerVolumeSpecName "kube-api-access-jhdt5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 13:58:42.737104 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:42.737037    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/878f435a-160a-4634-9ed2-e91c488b6854-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "878f435a-160a-4634-9ed2-e91c488b6854" (UID: "878f435a-160a-4634-9ed2-e91c488b6854"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 13:58:42.835704 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:42.835667    2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/878f435a-160a-4634-9ed2-e91c488b6854-oauth-serving-cert\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 13:58:42.835704 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:42.835706    2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/878f435a-160a-4634-9ed2-e91c488b6854-console-oauth-config\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 13:58:42.835878 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:42.835720    2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/878f435a-160a-4634-9ed2-e91c488b6854-console-serving-cert\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 13:58:42.835878 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:42.835735    2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/878f435a-160a-4634-9ed2-e91c488b6854-service-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 13:58:42.835878 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:42.835748    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhdt5\" (UniqueName: \"kubernetes.io/projected/878f435a-160a-4634-9ed2-e91c488b6854-kube-api-access-jhdt5\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 13:58:42.835878 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:42.835761    2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/878f435a-160a-4634-9ed2-e91c488b6854-console-config\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 13:58:43.371081 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:43.371045    2560 generic.go:356] "Generic (PLEG): container finished" podID="878f435a-160a-4634-9ed2-e91c488b6854" containerID="3d4032128f4155eb92089043d30cd830f0b15be0400551ba2125c133fc52d41a" exitCode=0
Sep 17 13:58:43.371506 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:43.371099    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fb686c6f6-58glc"
Sep 17 13:58:43.371506 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:43.371129    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fb686c6f6-58glc" event={"ID":"878f435a-160a-4634-9ed2-e91c488b6854","Type":"ContainerDied","Data":"3d4032128f4155eb92089043d30cd830f0b15be0400551ba2125c133fc52d41a"}
Sep 17 13:58:43.371506 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:43.371206    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fb686c6f6-58glc" event={"ID":"878f435a-160a-4634-9ed2-e91c488b6854","Type":"ContainerDied","Data":"c1360be142aa0ef729e3fd7fb57edae82a314190b702e99e897b94b93b73fa8e"}
Sep 17 13:58:43.371506 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:43.371226    2560 scope.go:117] "RemoveContainer" containerID="3d4032128f4155eb92089043d30cd830f0b15be0400551ba2125c133fc52d41a"
Sep 17 13:58:43.380602 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:43.380583    2560 scope.go:117] "RemoveContainer" containerID="3d4032128f4155eb92089043d30cd830f0b15be0400551ba2125c133fc52d41a"
Sep 17 13:58:43.381112 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:58:43.381054    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d4032128f4155eb92089043d30cd830f0b15be0400551ba2125c133fc52d41a\": container with ID starting with 3d4032128f4155eb92089043d30cd830f0b15be0400551ba2125c133fc52d41a not found: ID does not exist" containerID="3d4032128f4155eb92089043d30cd830f0b15be0400551ba2125c133fc52d41a"
Sep 17 13:58:43.381190 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:43.381098    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d4032128f4155eb92089043d30cd830f0b15be0400551ba2125c133fc52d41a"} err="failed to get container status \"3d4032128f4155eb92089043d30cd830f0b15be0400551ba2125c133fc52d41a\": rpc error: code = NotFound desc = could not find container \"3d4032128f4155eb92089043d30cd830f0b15be0400551ba2125c133fc52d41a\": container with ID starting with 3d4032128f4155eb92089043d30cd830f0b15be0400551ba2125c133fc52d41a not found: ID does not exist"
Sep 17 13:58:43.394662 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:43.394638    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-fb686c6f6-58glc"]
Sep 17 13:58:43.398392 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:43.398369    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-fb686c6f6-58glc"]
Sep 17 13:58:43.571236 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:43.571198    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="878f435a-160a-4634-9ed2-e91c488b6854" path="/var/lib/kubelet/pods/878f435a-160a-4634-9ed2-e91c488b6854/volumes"
Sep 17 13:58:58.523855 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:58.523814    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:58:58.625274 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:58:58.625237    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:59:18.918010 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:18.917975    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:59:19.194433 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:19.194351    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:59:20.014427 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:20.014396    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 13:59:20.014798 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:20.014683    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a29762f9-e0d5-43cd-b19f-2db732145b53" containerName="alertmanager" containerID="cri-o://5b21a340939172317b9e25530c46e14fb1f6e3c1bf3badb193c6a2257a230fb6" gracePeriod=120
Sep 17 13:59:20.014798 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:20.014759    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a29762f9-e0d5-43cd-b19f-2db732145b53" containerName="kube-rbac-proxy-web" containerID="cri-o://3fcaa8655428ec4e62d9404b4ec5ac72555b90b883987f9dd9b1e64910aa7e3f" gracePeriod=120
Sep 17 13:59:20.014798 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:20.014759    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a29762f9-e0d5-43cd-b19f-2db732145b53" containerName="kube-rbac-proxy-metric" containerID="cri-o://c67e8be4e049bd64c8436bb921e295fc72ce7c6542b8d55aa8b0ec847bf36ce8" gracePeriod=120
Sep 17 13:59:20.014965 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:20.014795    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a29762f9-e0d5-43cd-b19f-2db732145b53" containerName="kube-rbac-proxy" containerID="cri-o://681dd32d424c8518284f25f74946ea91eccd7a3c8b8f125de8ffeb7f6a7149cb" gracePeriod=120
Sep 17 13:59:20.014965 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:20.014828    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a29762f9-e0d5-43cd-b19f-2db732145b53" containerName="prom-label-proxy" containerID="cri-o://b6b6e03dc4d2cc5227a5d40f01b0fb59984f1b058483c35d37b41869d5e846ab" gracePeriod=120
Sep 17 13:59:20.014965 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:20.014843    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a29762f9-e0d5-43cd-b19f-2db732145b53" containerName="config-reloader" containerID="cri-o://70d6616d4f35e34922b8c01a6195f2e3e7b167223a4a75ca01ad18f46b4c7734" gracePeriod=120
Sep 17 13:59:20.520471 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:20.520439    2560 generic.go:356] "Generic (PLEG): container finished" podID="a29762f9-e0d5-43cd-b19f-2db732145b53" containerID="b6b6e03dc4d2cc5227a5d40f01b0fb59984f1b058483c35d37b41869d5e846ab" exitCode=0
Sep 17 13:59:20.520471 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:20.520483    2560 generic.go:356] "Generic (PLEG): container finished" podID="a29762f9-e0d5-43cd-b19f-2db732145b53" containerID="c67e8be4e049bd64c8436bb921e295fc72ce7c6542b8d55aa8b0ec847bf36ce8" exitCode=0
Sep 17 13:59:20.520684 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:20.520486    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a29762f9-e0d5-43cd-b19f-2db732145b53","Type":"ContainerDied","Data":"b6b6e03dc4d2cc5227a5d40f01b0fb59984f1b058483c35d37b41869d5e846ab"}
Sep 17 13:59:20.520684 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:20.520550    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a29762f9-e0d5-43cd-b19f-2db732145b53","Type":"ContainerDied","Data":"c67e8be4e049bd64c8436bb921e295fc72ce7c6542b8d55aa8b0ec847bf36ce8"}
Sep 17 13:59:20.520684 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:20.520495    2560 generic.go:356] "Generic (PLEG): container finished" podID="a29762f9-e0d5-43cd-b19f-2db732145b53" containerID="681dd32d424c8518284f25f74946ea91eccd7a3c8b8f125de8ffeb7f6a7149cb" exitCode=0
Sep 17 13:59:20.520684 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:20.520583    2560 generic.go:356] "Generic (PLEG): container finished" podID="a29762f9-e0d5-43cd-b19f-2db732145b53" containerID="70d6616d4f35e34922b8c01a6195f2e3e7b167223a4a75ca01ad18f46b4c7734" exitCode=0
Sep 17 13:59:20.520684 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:20.520591    2560 generic.go:356] "Generic (PLEG): container finished" podID="a29762f9-e0d5-43cd-b19f-2db732145b53" containerID="5b21a340939172317b9e25530c46e14fb1f6e3c1bf3badb193c6a2257a230fb6" exitCode=0
Sep 17 13:59:20.520684 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:20.520576    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a29762f9-e0d5-43cd-b19f-2db732145b53","Type":"ContainerDied","Data":"681dd32d424c8518284f25f74946ea91eccd7a3c8b8f125de8ffeb7f6a7149cb"}
Sep 17 13:59:20.520863 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:20.520695    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a29762f9-e0d5-43cd-b19f-2db732145b53","Type":"ContainerDied","Data":"70d6616d4f35e34922b8c01a6195f2e3e7b167223a4a75ca01ad18f46b4c7734"}
Sep 17 13:59:20.520863 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:20.520731    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a29762f9-e0d5-43cd-b19f-2db732145b53","Type":"ContainerDied","Data":"5b21a340939172317b9e25530c46e14fb1f6e3c1bf3badb193c6a2257a230fb6"}
Sep 17 13:59:21.258132 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.258108    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:59:21.312844 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.309677    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a29762f9-e0d5-43cd-b19f-2db732145b53-secret-alertmanager-main-tls\") pod \"a29762f9-e0d5-43cd-b19f-2db732145b53\" (UID: \"a29762f9-e0d5-43cd-b19f-2db732145b53\") "
Sep 17 13:59:21.312844 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.309720    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a29762f9-e0d5-43cd-b19f-2db732145b53-cluster-tls-config\") pod \"a29762f9-e0d5-43cd-b19f-2db732145b53\" (UID: \"a29762f9-e0d5-43cd-b19f-2db732145b53\") "
Sep 17 13:59:21.312844 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.309751    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a29762f9-e0d5-43cd-b19f-2db732145b53-secret-alertmanager-kube-rbac-proxy-web\") pod \"a29762f9-e0d5-43cd-b19f-2db732145b53\" (UID: \"a29762f9-e0d5-43cd-b19f-2db732145b53\") "
Sep 17 13:59:21.312844 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.309821    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a29762f9-e0d5-43cd-b19f-2db732145b53-alertmanager-main-db\") pod \"a29762f9-e0d5-43cd-b19f-2db732145b53\" (UID: \"a29762f9-e0d5-43cd-b19f-2db732145b53\") "
Sep 17 13:59:21.312844 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.309875    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a29762f9-e0d5-43cd-b19f-2db732145b53-secret-alertmanager-kube-rbac-proxy\") pod \"a29762f9-e0d5-43cd-b19f-2db732145b53\" (UID: \"a29762f9-e0d5-43cd-b19f-2db732145b53\") "
Sep 17 13:59:21.312844 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.309903    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a29762f9-e0d5-43cd-b19f-2db732145b53-metrics-client-ca\") pod \"a29762f9-e0d5-43cd-b19f-2db732145b53\" (UID: \"a29762f9-e0d5-43cd-b19f-2db732145b53\") "
Sep 17 13:59:21.312844 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.309955    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a29762f9-e0d5-43cd-b19f-2db732145b53-web-config\") pod \"a29762f9-e0d5-43cd-b19f-2db732145b53\" (UID: \"a29762f9-e0d5-43cd-b19f-2db732145b53\") "
Sep 17 13:59:21.312844 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.309992    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a29762f9-e0d5-43cd-b19f-2db732145b53-config-volume\") pod \"a29762f9-e0d5-43cd-b19f-2db732145b53\" (UID: \"a29762f9-e0d5-43cd-b19f-2db732145b53\") "
Sep 17 13:59:21.312844 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.310021    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a29762f9-e0d5-43cd-b19f-2db732145b53-config-out\") pod \"a29762f9-e0d5-43cd-b19f-2db732145b53\" (UID: \"a29762f9-e0d5-43cd-b19f-2db732145b53\") "
Sep 17 13:59:21.312844 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.310044    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a29762f9-e0d5-43cd-b19f-2db732145b53-tls-assets\") pod \"a29762f9-e0d5-43cd-b19f-2db732145b53\" (UID: \"a29762f9-e0d5-43cd-b19f-2db732145b53\") "
Sep 17 13:59:21.312844 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.310089    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2lnq4\" (UniqueName: \"kubernetes.io/projected/a29762f9-e0d5-43cd-b19f-2db732145b53-kube-api-access-2lnq4\") pod \"a29762f9-e0d5-43cd-b19f-2db732145b53\" (UID: \"a29762f9-e0d5-43cd-b19f-2db732145b53\") "
Sep 17 13:59:21.312844 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.310124    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a29762f9-e0d5-43cd-b19f-2db732145b53-secret-alertmanager-kube-rbac-proxy-metric\") pod \"a29762f9-e0d5-43cd-b19f-2db732145b53\" (UID: \"a29762f9-e0d5-43cd-b19f-2db732145b53\") "
Sep 17 13:59:21.312844 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.310152    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a29762f9-e0d5-43cd-b19f-2db732145b53-alertmanager-trusted-ca-bundle\") pod \"a29762f9-e0d5-43cd-b19f-2db732145b53\" (UID: \"a29762f9-e0d5-43cd-b19f-2db732145b53\") "
Sep 17 13:59:21.312844 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.310745    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a29762f9-e0d5-43cd-b19f-2db732145b53-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "a29762f9-e0d5-43cd-b19f-2db732145b53" (UID: "a29762f9-e0d5-43cd-b19f-2db732145b53"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 13:59:21.313774 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.313422    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a29762f9-e0d5-43cd-b19f-2db732145b53-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "a29762f9-e0d5-43cd-b19f-2db732145b53" (UID: "a29762f9-e0d5-43cd-b19f-2db732145b53"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 13:59:21.316643 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.316610    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a29762f9-e0d5-43cd-b19f-2db732145b53-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "a29762f9-e0d5-43cd-b19f-2db732145b53" (UID: "a29762f9-e0d5-43cd-b19f-2db732145b53"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 13:59:21.316932 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.316908    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a29762f9-e0d5-43cd-b19f-2db732145b53-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "a29762f9-e0d5-43cd-b19f-2db732145b53" (UID: "a29762f9-e0d5-43cd-b19f-2db732145b53"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 13:59:21.316932 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.316919    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a29762f9-e0d5-43cd-b19f-2db732145b53-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "a29762f9-e0d5-43cd-b19f-2db732145b53" (UID: "a29762f9-e0d5-43cd-b19f-2db732145b53"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 13:59:21.317572 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.317526    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a29762f9-e0d5-43cd-b19f-2db732145b53-config-volume" (OuterVolumeSpecName: "config-volume") pod "a29762f9-e0d5-43cd-b19f-2db732145b53" (UID: "a29762f9-e0d5-43cd-b19f-2db732145b53"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 13:59:21.319223 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.318285    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a29762f9-e0d5-43cd-b19f-2db732145b53-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "a29762f9-e0d5-43cd-b19f-2db732145b53" (UID: "a29762f9-e0d5-43cd-b19f-2db732145b53"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 13:59:21.319223 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.318384    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a29762f9-e0d5-43cd-b19f-2db732145b53-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "a29762f9-e0d5-43cd-b19f-2db732145b53" (UID: "a29762f9-e0d5-43cd-b19f-2db732145b53"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 13:59:21.319223 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.318595    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a29762f9-e0d5-43cd-b19f-2db732145b53-config-out" (OuterVolumeSpecName: "config-out") pod "a29762f9-e0d5-43cd-b19f-2db732145b53" (UID: "a29762f9-e0d5-43cd-b19f-2db732145b53"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 13:59:21.319223 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.318986    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a29762f9-e0d5-43cd-b19f-2db732145b53-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "a29762f9-e0d5-43cd-b19f-2db732145b53" (UID: "a29762f9-e0d5-43cd-b19f-2db732145b53"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 13:59:21.319527 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.319504    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a29762f9-e0d5-43cd-b19f-2db732145b53-kube-api-access-2lnq4" (OuterVolumeSpecName: "kube-api-access-2lnq4") pod "a29762f9-e0d5-43cd-b19f-2db732145b53" (UID: "a29762f9-e0d5-43cd-b19f-2db732145b53"). InnerVolumeSpecName "kube-api-access-2lnq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 13:59:21.322174 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.322091    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a29762f9-e0d5-43cd-b19f-2db732145b53-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "a29762f9-e0d5-43cd-b19f-2db732145b53" (UID: "a29762f9-e0d5-43cd-b19f-2db732145b53"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 13:59:21.329497 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.329443    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a29762f9-e0d5-43cd-b19f-2db732145b53-web-config" (OuterVolumeSpecName: "web-config") pod "a29762f9-e0d5-43cd-b19f-2db732145b53" (UID: "a29762f9-e0d5-43cd-b19f-2db732145b53"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 13:59:21.410962 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.410882    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lnq4\" (UniqueName: \"kubernetes.io/projected/a29762f9-e0d5-43cd-b19f-2db732145b53-kube-api-access-2lnq4\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 13:59:21.410962 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.410917    2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a29762f9-e0d5-43cd-b19f-2db732145b53-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 13:59:21.410962 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.410928    2560 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a29762f9-e0d5-43cd-b19f-2db732145b53-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 13:59:21.410962 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.410937    2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a29762f9-e0d5-43cd-b19f-2db732145b53-secret-alertmanager-main-tls\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 13:59:21.410962 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.410945    2560 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a29762f9-e0d5-43cd-b19f-2db732145b53-cluster-tls-config\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 13:59:21.410962 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.410953    2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a29762f9-e0d5-43cd-b19f-2db732145b53-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 13:59:21.410962 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.410961    2560 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a29762f9-e0d5-43cd-b19f-2db732145b53-alertmanager-main-db\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 13:59:21.410962 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.410970    2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a29762f9-e0d5-43cd-b19f-2db732145b53-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 13:59:21.410962 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.410978    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a29762f9-e0d5-43cd-b19f-2db732145b53-metrics-client-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 13:59:21.411357 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.410987    2560 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a29762f9-e0d5-43cd-b19f-2db732145b53-web-config\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 13:59:21.411357 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.410995    2560 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a29762f9-e0d5-43cd-b19f-2db732145b53-config-volume\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 13:59:21.411357 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.411002    2560 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a29762f9-e0d5-43cd-b19f-2db732145b53-config-out\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 13:59:21.411357 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.411009    2560 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a29762f9-e0d5-43cd-b19f-2db732145b53-tls-assets\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 13:59:21.533666 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.533624    2560 generic.go:356] "Generic (PLEG): container finished" podID="a29762f9-e0d5-43cd-b19f-2db732145b53" containerID="3fcaa8655428ec4e62d9404b4ec5ac72555b90b883987f9dd9b1e64910aa7e3f" exitCode=0
Sep 17 13:59:21.533822 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.533723    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:59:21.533822 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.533730    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a29762f9-e0d5-43cd-b19f-2db732145b53","Type":"ContainerDied","Data":"3fcaa8655428ec4e62d9404b4ec5ac72555b90b883987f9dd9b1e64910aa7e3f"}
Sep 17 13:59:21.533904 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.533836    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a29762f9-e0d5-43cd-b19f-2db732145b53","Type":"ContainerDied","Data":"3bfcdc3744fe01b27fc183c70fec18087250212c11ccc0bd1d1a8f685a77f98f"}
Sep 17 13:59:21.533904 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.533883    2560 scope.go:117] "RemoveContainer" containerID="b6b6e03dc4d2cc5227a5d40f01b0fb59984f1b058483c35d37b41869d5e846ab"
Sep 17 13:59:21.542797 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.542779    2560 scope.go:117] "RemoveContainer" containerID="c67e8be4e049bd64c8436bb921e295fc72ce7c6542b8d55aa8b0ec847bf36ce8"
Sep 17 13:59:21.550925 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.550905    2560 scope.go:117] "RemoveContainer" containerID="681dd32d424c8518284f25f74946ea91eccd7a3c8b8f125de8ffeb7f6a7149cb"
Sep 17 13:59:21.558650 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.558630    2560 scope.go:117] "RemoveContainer" containerID="3fcaa8655428ec4e62d9404b4ec5ac72555b90b883987f9dd9b1e64910aa7e3f"
Sep 17 13:59:21.568901 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.568830    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 13:59:21.568901 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.568888    2560 scope.go:117] "RemoveContainer" containerID="70d6616d4f35e34922b8c01a6195f2e3e7b167223a4a75ca01ad18f46b4c7734"
Sep 17 13:59:21.578342 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.578314    2560 scope.go:117] "RemoveContainer" containerID="5b21a340939172317b9e25530c46e14fb1f6e3c1bf3badb193c6a2257a230fb6"
Sep 17 13:59:21.578571 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.578541    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 13:59:21.586067 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.586047    2560 scope.go:117] "RemoveContainer" containerID="f43b8b6cb2c544eb1cf4de34c863c18f03b94303fac98bf70955d30062fe1f51"
Sep 17 13:59:21.592855 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.592841    2560 scope.go:117] "RemoveContainer" containerID="b6b6e03dc4d2cc5227a5d40f01b0fb59984f1b058483c35d37b41869d5e846ab"
Sep 17 13:59:21.593068 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:59:21.593051    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6b6e03dc4d2cc5227a5d40f01b0fb59984f1b058483c35d37b41869d5e846ab\": container with ID starting with b6b6e03dc4d2cc5227a5d40f01b0fb59984f1b058483c35d37b41869d5e846ab not found: ID does not exist" containerID="b6b6e03dc4d2cc5227a5d40f01b0fb59984f1b058483c35d37b41869d5e846ab"
Sep 17 13:59:21.593126 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.593077    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6b6e03dc4d2cc5227a5d40f01b0fb59984f1b058483c35d37b41869d5e846ab"} err="failed to get container status \"b6b6e03dc4d2cc5227a5d40f01b0fb59984f1b058483c35d37b41869d5e846ab\": rpc error: code = NotFound desc = could not find container \"b6b6e03dc4d2cc5227a5d40f01b0fb59984f1b058483c35d37b41869d5e846ab\": container with ID starting with b6b6e03dc4d2cc5227a5d40f01b0fb59984f1b058483c35d37b41869d5e846ab not found: ID does not exist"
Sep 17 13:59:21.593126 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.593100    2560 scope.go:117] "RemoveContainer" containerID="c67e8be4e049bd64c8436bb921e295fc72ce7c6542b8d55aa8b0ec847bf36ce8"
Sep 17 13:59:21.593369 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:59:21.593337    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c67e8be4e049bd64c8436bb921e295fc72ce7c6542b8d55aa8b0ec847bf36ce8\": container with ID starting with c67e8be4e049bd64c8436bb921e295fc72ce7c6542b8d55aa8b0ec847bf36ce8 not found: ID does not exist" containerID="c67e8be4e049bd64c8436bb921e295fc72ce7c6542b8d55aa8b0ec847bf36ce8"
Sep 17 13:59:21.593424 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.593374    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c67e8be4e049bd64c8436bb921e295fc72ce7c6542b8d55aa8b0ec847bf36ce8"} err="failed to get container status \"c67e8be4e049bd64c8436bb921e295fc72ce7c6542b8d55aa8b0ec847bf36ce8\": rpc error: code = NotFound desc = could not find container \"c67e8be4e049bd64c8436bb921e295fc72ce7c6542b8d55aa8b0ec847bf36ce8\": container with ID starting with c67e8be4e049bd64c8436bb921e295fc72ce7c6542b8d55aa8b0ec847bf36ce8 not found: ID does not exist"
Sep 17 13:59:21.593424 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.593390    2560 scope.go:117] "RemoveContainer" containerID="681dd32d424c8518284f25f74946ea91eccd7a3c8b8f125de8ffeb7f6a7149cb"
Sep 17 13:59:21.593652 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:59:21.593634    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"681dd32d424c8518284f25f74946ea91eccd7a3c8b8f125de8ffeb7f6a7149cb\": container with ID starting with 681dd32d424c8518284f25f74946ea91eccd7a3c8b8f125de8ffeb7f6a7149cb not found: ID does not exist" containerID="681dd32d424c8518284f25f74946ea91eccd7a3c8b8f125de8ffeb7f6a7149cb"
Sep 17 13:59:21.593707 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.593655    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"681dd32d424c8518284f25f74946ea91eccd7a3c8b8f125de8ffeb7f6a7149cb"} err="failed to get container status \"681dd32d424c8518284f25f74946ea91eccd7a3c8b8f125de8ffeb7f6a7149cb\": rpc error: code = NotFound desc = could not find container \"681dd32d424c8518284f25f74946ea91eccd7a3c8b8f125de8ffeb7f6a7149cb\": container with ID starting with 681dd32d424c8518284f25f74946ea91eccd7a3c8b8f125de8ffeb7f6a7149cb not found: ID does not exist"
Sep 17 13:59:21.593707 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.593669    2560 scope.go:117] "RemoveContainer" containerID="3fcaa8655428ec4e62d9404b4ec5ac72555b90b883987f9dd9b1e64910aa7e3f"
Sep 17 13:59:21.593895 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:59:21.593878    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fcaa8655428ec4e62d9404b4ec5ac72555b90b883987f9dd9b1e64910aa7e3f\": container with ID starting with 3fcaa8655428ec4e62d9404b4ec5ac72555b90b883987f9dd9b1e64910aa7e3f not found: ID does not exist" containerID="3fcaa8655428ec4e62d9404b4ec5ac72555b90b883987f9dd9b1e64910aa7e3f"
Sep 17 13:59:21.593950 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.593897    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fcaa8655428ec4e62d9404b4ec5ac72555b90b883987f9dd9b1e64910aa7e3f"} err="failed to get container status \"3fcaa8655428ec4e62d9404b4ec5ac72555b90b883987f9dd9b1e64910aa7e3f\": rpc error: code = NotFound desc = could not find container \"3fcaa8655428ec4e62d9404b4ec5ac72555b90b883987f9dd9b1e64910aa7e3f\": container with ID starting with 3fcaa8655428ec4e62d9404b4ec5ac72555b90b883987f9dd9b1e64910aa7e3f not found: ID does not exist"
Sep 17 13:59:21.593950 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.593910    2560 scope.go:117] "RemoveContainer" containerID="70d6616d4f35e34922b8c01a6195f2e3e7b167223a4a75ca01ad18f46b4c7734"
Sep 17 13:59:21.594122 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:59:21.594108    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70d6616d4f35e34922b8c01a6195f2e3e7b167223a4a75ca01ad18f46b4c7734\": container with ID starting with 70d6616d4f35e34922b8c01a6195f2e3e7b167223a4a75ca01ad18f46b4c7734 not found: ID does not exist" containerID="70d6616d4f35e34922b8c01a6195f2e3e7b167223a4a75ca01ad18f46b4c7734"
Sep 17 13:59:21.594178 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.594126    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70d6616d4f35e34922b8c01a6195f2e3e7b167223a4a75ca01ad18f46b4c7734"} err="failed to get container status \"70d6616d4f35e34922b8c01a6195f2e3e7b167223a4a75ca01ad18f46b4c7734\": rpc error: code = NotFound desc = could not find container \"70d6616d4f35e34922b8c01a6195f2e3e7b167223a4a75ca01ad18f46b4c7734\": container with ID starting with 70d6616d4f35e34922b8c01a6195f2e3e7b167223a4a75ca01ad18f46b4c7734 not found: ID does not exist"
Sep 17 13:59:21.594178 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.594137    2560 scope.go:117] "RemoveContainer" containerID="5b21a340939172317b9e25530c46e14fb1f6e3c1bf3badb193c6a2257a230fb6"
Sep 17 13:59:21.594361 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:59:21.594345    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b21a340939172317b9e25530c46e14fb1f6e3c1bf3badb193c6a2257a230fb6\": container with ID starting with 5b21a340939172317b9e25530c46e14fb1f6e3c1bf3badb193c6a2257a230fb6 not found: ID does not exist" containerID="5b21a340939172317b9e25530c46e14fb1f6e3c1bf3badb193c6a2257a230fb6"
Sep 17 13:59:21.594411 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.594365    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b21a340939172317b9e25530c46e14fb1f6e3c1bf3badb193c6a2257a230fb6"} err="failed to get container status \"5b21a340939172317b9e25530c46e14fb1f6e3c1bf3badb193c6a2257a230fb6\": rpc error: code = NotFound desc = could not find container \"5b21a340939172317b9e25530c46e14fb1f6e3c1bf3badb193c6a2257a230fb6\": container with ID starting with 5b21a340939172317b9e25530c46e14fb1f6e3c1bf3badb193c6a2257a230fb6 not found: ID does not exist"
Sep 17 13:59:21.594411 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.594380    2560 scope.go:117] "RemoveContainer" containerID="f43b8b6cb2c544eb1cf4de34c863c18f03b94303fac98bf70955d30062fe1f51"
Sep 17 13:59:21.594605 ip-10-0-129-158 kubenswrapper[2560]: E0917 13:59:21.594585    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f43b8b6cb2c544eb1cf4de34c863c18f03b94303fac98bf70955d30062fe1f51\": container with ID starting with f43b8b6cb2c544eb1cf4de34c863c18f03b94303fac98bf70955d30062fe1f51 not found: ID does not exist" containerID="f43b8b6cb2c544eb1cf4de34c863c18f03b94303fac98bf70955d30062fe1f51"
Sep 17 13:59:21.594655 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.594608    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f43b8b6cb2c544eb1cf4de34c863c18f03b94303fac98bf70955d30062fe1f51"} err="failed to get container status \"f43b8b6cb2c544eb1cf4de34c863c18f03b94303fac98bf70955d30062fe1f51\": rpc error: code = NotFound desc = could not find container \"f43b8b6cb2c544eb1cf4de34c863c18f03b94303fac98bf70955d30062fe1f51\": container with ID starting with f43b8b6cb2c544eb1cf4de34c863c18f03b94303fac98bf70955d30062fe1f51 not found: ID does not exist"
Sep 17 13:59:21.617418 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.613507    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 13:59:21.617418 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.614143    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="878f435a-160a-4634-9ed2-e91c488b6854" containerName="console"
Sep 17 13:59:21.617418 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.614160    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="878f435a-160a-4634-9ed2-e91c488b6854" containerName="console"
Sep 17 13:59:21.617418 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.614179    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a29762f9-e0d5-43cd-b19f-2db732145b53" containerName="kube-rbac-proxy-web"
Sep 17 13:59:21.617418 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.614189    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a29762f9-e0d5-43cd-b19f-2db732145b53" containerName="kube-rbac-proxy-web"
Sep 17 13:59:21.617418 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.614202    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a29762f9-e0d5-43cd-b19f-2db732145b53" containerName="kube-rbac-proxy"
Sep 17 13:59:21.617418 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.614210    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a29762f9-e0d5-43cd-b19f-2db732145b53" containerName="kube-rbac-proxy"
Sep 17 13:59:21.617418 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.614227    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a29762f9-e0d5-43cd-b19f-2db732145b53" containerName="prom-label-proxy"
Sep 17 13:59:21.617418 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.614235    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a29762f9-e0d5-43cd-b19f-2db732145b53" containerName="prom-label-proxy"
Sep 17 13:59:21.617418 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.614256    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a29762f9-e0d5-43cd-b19f-2db732145b53" containerName="config-reloader"
Sep 17 13:59:21.617418 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.614264    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a29762f9-e0d5-43cd-b19f-2db732145b53" containerName="config-reloader"
Sep 17 13:59:21.617418 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.614287    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a29762f9-e0d5-43cd-b19f-2db732145b53" containerName="init-config-reloader"
Sep 17 13:59:21.617418 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.614296    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a29762f9-e0d5-43cd-b19f-2db732145b53" containerName="init-config-reloader"
Sep 17 13:59:21.617418 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.614320    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a29762f9-e0d5-43cd-b19f-2db732145b53" containerName="kube-rbac-proxy-metric"
Sep 17 13:59:21.617418 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.614328    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a29762f9-e0d5-43cd-b19f-2db732145b53" containerName="kube-rbac-proxy-metric"
Sep 17 13:59:21.617418 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.614351    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a29762f9-e0d5-43cd-b19f-2db732145b53" containerName="alertmanager"
Sep 17 13:59:21.617418 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.614359    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a29762f9-e0d5-43cd-b19f-2db732145b53" containerName="alertmanager"
Sep 17 13:59:21.617418 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.614503    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a29762f9-e0d5-43cd-b19f-2db732145b53" containerName="kube-rbac-proxy"
Sep 17 13:59:21.617418 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.614513    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a29762f9-e0d5-43cd-b19f-2db732145b53" containerName="prom-label-proxy"
Sep 17 13:59:21.617418 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.614524    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a29762f9-e0d5-43cd-b19f-2db732145b53" containerName="kube-rbac-proxy-metric"
Sep 17 13:59:21.617418 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.614532    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a29762f9-e0d5-43cd-b19f-2db732145b53" containerName="config-reloader"
Sep 17 13:59:21.617418 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.614547    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a29762f9-e0d5-43cd-b19f-2db732145b53" containerName="alertmanager"
Sep 17 13:59:21.617418 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.614555    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="878f435a-160a-4634-9ed2-e91c488b6854" containerName="console"
Sep 17 13:59:21.617418 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.614572    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a29762f9-e0d5-43cd-b19f-2db732145b53" containerName="kube-rbac-proxy-web"
Sep 17 13:59:21.621858 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.621835    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:59:21.625126 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.624878    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 17 13:59:21.625126 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.624991    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 17 13:59:21.626156 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.626132    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 17 13:59:21.626745 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.626438    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 17 13:59:21.626745 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.626482    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 17 13:59:21.626745 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.626640    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 17 13:59:21.626956 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.626875    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 17 13:59:21.627006 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.626973    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 17 13:59:21.627166 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.627137    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-cqm8j\""
Sep 17 13:59:21.631169 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.631141    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 17 13:59:21.632955 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.632932    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 13:59:21.712329 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.712246    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0327e6ab-d63a-438a-b925-fc60d270bc4d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0327e6ab-d63a-438a-b925-fc60d270bc4d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:59:21.712329 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.712280    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0327e6ab-d63a-438a-b925-fc60d270bc4d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0327e6ab-d63a-438a-b925-fc60d270bc4d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:59:21.712329 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.712297    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8ckn\" (UniqueName: \"kubernetes.io/projected/0327e6ab-d63a-438a-b925-fc60d270bc4d-kube-api-access-b8ckn\") pod \"alertmanager-main-0\" (UID: \"0327e6ab-d63a-438a-b925-fc60d270bc4d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:59:21.712596 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.712372    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0327e6ab-d63a-438a-b925-fc60d270bc4d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0327e6ab-d63a-438a-b925-fc60d270bc4d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:59:21.712596 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.712409    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0327e6ab-d63a-438a-b925-fc60d270bc4d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0327e6ab-d63a-438a-b925-fc60d270bc4d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:59:21.712596 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.712524    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0327e6ab-d63a-438a-b925-fc60d270bc4d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0327e6ab-d63a-438a-b925-fc60d270bc4d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:59:21.712596 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.712562    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0327e6ab-d63a-438a-b925-fc60d270bc4d-config-volume\") pod \"alertmanager-main-0\" (UID: \"0327e6ab-d63a-438a-b925-fc60d270bc4d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:59:21.712596 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.712584    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0327e6ab-d63a-438a-b925-fc60d270bc4d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0327e6ab-d63a-438a-b925-fc60d270bc4d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:59:21.712784 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.712602    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0327e6ab-d63a-438a-b925-fc60d270bc4d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0327e6ab-d63a-438a-b925-fc60d270bc4d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:59:21.712784 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.712621    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0327e6ab-d63a-438a-b925-fc60d270bc4d-config-out\") pod \"alertmanager-main-0\" (UID: \"0327e6ab-d63a-438a-b925-fc60d270bc4d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:59:21.712784 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.712700    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0327e6ab-d63a-438a-b925-fc60d270bc4d-web-config\") pod \"alertmanager-main-0\" (UID: \"0327e6ab-d63a-438a-b925-fc60d270bc4d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:59:21.712784 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.712728    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0327e6ab-d63a-438a-b925-fc60d270bc4d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0327e6ab-d63a-438a-b925-fc60d270bc4d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:59:21.712784 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.712751    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0327e6ab-d63a-438a-b925-fc60d270bc4d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0327e6ab-d63a-438a-b925-fc60d270bc4d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:59:21.814018 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.813965    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0327e6ab-d63a-438a-b925-fc60d270bc4d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0327e6ab-d63a-438a-b925-fc60d270bc4d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:59:21.814215 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.814031    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0327e6ab-d63a-438a-b925-fc60d270bc4d-config-volume\") pod \"alertmanager-main-0\" (UID: \"0327e6ab-d63a-438a-b925-fc60d270bc4d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:59:21.814215 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.814059    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0327e6ab-d63a-438a-b925-fc60d270bc4d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0327e6ab-d63a-438a-b925-fc60d270bc4d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:59:21.814215 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.814157    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0327e6ab-d63a-438a-b925-fc60d270bc4d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0327e6ab-d63a-438a-b925-fc60d270bc4d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:59:21.814215 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.814206    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0327e6ab-d63a-438a-b925-fc60d270bc4d-config-out\") pod \"alertmanager-main-0\" (UID: \"0327e6ab-d63a-438a-b925-fc60d270bc4d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:59:21.814419 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.814236    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0327e6ab-d63a-438a-b925-fc60d270bc4d-web-config\") pod \"alertmanager-main-0\" (UID: \"0327e6ab-d63a-438a-b925-fc60d270bc4d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:59:21.814419 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.814266    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0327e6ab-d63a-438a-b925-fc60d270bc4d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0327e6ab-d63a-438a-b925-fc60d270bc4d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:59:21.814419 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.814400    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0327e6ab-d63a-438a-b925-fc60d270bc4d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0327e6ab-d63a-438a-b925-fc60d270bc4d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:59:21.814601 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.814438    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0327e6ab-d63a-438a-b925-fc60d270bc4d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0327e6ab-d63a-438a-b925-fc60d270bc4d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:59:21.814601 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.814485    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0327e6ab-d63a-438a-b925-fc60d270bc4d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0327e6ab-d63a-438a-b925-fc60d270bc4d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:59:21.814601 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.814512    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8ckn\" (UniqueName: \"kubernetes.io/projected/0327e6ab-d63a-438a-b925-fc60d270bc4d-kube-api-access-b8ckn\") pod \"alertmanager-main-0\" (UID: \"0327e6ab-d63a-438a-b925-fc60d270bc4d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:59:21.814601 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.814548    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0327e6ab-d63a-438a-b925-fc60d270bc4d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0327e6ab-d63a-438a-b925-fc60d270bc4d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:59:21.814601 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.814574    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0327e6ab-d63a-438a-b925-fc60d270bc4d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0327e6ab-d63a-438a-b925-fc60d270bc4d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:59:21.814915 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.814894    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0327e6ab-d63a-438a-b925-fc60d270bc4d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0327e6ab-d63a-438a-b925-fc60d270bc4d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:59:21.814979 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.814949    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0327e6ab-d63a-438a-b925-fc60d270bc4d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0327e6ab-d63a-438a-b925-fc60d270bc4d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:59:21.816949 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.816918    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0327e6ab-d63a-438a-b925-fc60d270bc4d-config-out\") pod \"alertmanager-main-0\" (UID: \"0327e6ab-d63a-438a-b925-fc60d270bc4d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:59:21.817522 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.817099    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0327e6ab-d63a-438a-b925-fc60d270bc4d-config-volume\") pod \"alertmanager-main-0\" (UID: \"0327e6ab-d63a-438a-b925-fc60d270bc4d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:59:21.817522 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.817197    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0327e6ab-d63a-438a-b925-fc60d270bc4d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0327e6ab-d63a-438a-b925-fc60d270bc4d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:59:21.817522 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.817403    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0327e6ab-d63a-438a-b925-fc60d270bc4d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0327e6ab-d63a-438a-b925-fc60d270bc4d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:59:21.817522 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.817454    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0327e6ab-d63a-438a-b925-fc60d270bc4d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0327e6ab-d63a-438a-b925-fc60d270bc4d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:59:21.817776 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.817598    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0327e6ab-d63a-438a-b925-fc60d270bc4d-web-config\") pod \"alertmanager-main-0\" (UID: \"0327e6ab-d63a-438a-b925-fc60d270bc4d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:59:21.817776 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.817644    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0327e6ab-d63a-438a-b925-fc60d270bc4d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0327e6ab-d63a-438a-b925-fc60d270bc4d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:59:21.817776 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.817671    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0327e6ab-d63a-438a-b925-fc60d270bc4d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0327e6ab-d63a-438a-b925-fc60d270bc4d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:59:21.817969 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.817950    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0327e6ab-d63a-438a-b925-fc60d270bc4d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0327e6ab-d63a-438a-b925-fc60d270bc4d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:59:21.818869 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.818852    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0327e6ab-d63a-438a-b925-fc60d270bc4d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0327e6ab-d63a-438a-b925-fc60d270bc4d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:59:21.831762 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.831746    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8ckn\" (UniqueName: \"kubernetes.io/projected/0327e6ab-d63a-438a-b925-fc60d270bc4d-kube-api-access-b8ckn\") pod \"alertmanager-main-0\" (UID: \"0327e6ab-d63a-438a-b925-fc60d270bc4d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:59:21.934591 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:21.934563    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 13:59:22.214141 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:22.214106    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 13:59:22.217772 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:59:22.217735    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0327e6ab_d63a_438a_b925_fc60d270bc4d.slice/crio-1c5ad6b704caa0dad544ddfd72e5adb72309f32cc8d19be1c6de15b7580c89c9 WatchSource:0}: Error finding container 1c5ad6b704caa0dad544ddfd72e5adb72309f32cc8d19be1c6de15b7580c89c9: Status 404 returned error can't find the container with id 1c5ad6b704caa0dad544ddfd72e5adb72309f32cc8d19be1c6de15b7580c89c9
Sep 17 13:59:22.540236 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:22.540158    2560 generic.go:356] "Generic (PLEG): container finished" podID="0327e6ab-d63a-438a-b925-fc60d270bc4d" containerID="76725610afc74746abdda799442b3c0e552eac6c0512c745802f85afb206a3a7" exitCode=0
Sep 17 13:59:22.540236 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:22.540194    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0327e6ab-d63a-438a-b925-fc60d270bc4d","Type":"ContainerDied","Data":"76725610afc74746abdda799442b3c0e552eac6c0512c745802f85afb206a3a7"}
Sep 17 13:59:22.540590 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:22.540251    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0327e6ab-d63a-438a-b925-fc60d270bc4d","Type":"ContainerStarted","Data":"1c5ad6b704caa0dad544ddfd72e5adb72309f32cc8d19be1c6de15b7580c89c9"}
Sep 17 13:59:23.547405 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:23.547324    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0327e6ab-d63a-438a-b925-fc60d270bc4d","Type":"ContainerStarted","Data":"1d09d9ae1eb279f4e0f921c655204b69fe222cb3fa26422a4d2c7b5d1b339d22"}
Sep 17 13:59:23.547405 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:23.547397    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0327e6ab-d63a-438a-b925-fc60d270bc4d","Type":"ContainerStarted","Data":"d14f1b72841cdf0392a02a3c39c36f4bdcc66d4dee3c5e69b66305a66d492b56"}
Sep 17 13:59:23.547914 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:23.547440    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0327e6ab-d63a-438a-b925-fc60d270bc4d","Type":"ContainerStarted","Data":"222226c72e08868bbe28a535f98c575fedb05b73787d8b2a03655bc4b2a3bad6"}
Sep 17 13:59:23.547914 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:23.547496    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0327e6ab-d63a-438a-b925-fc60d270bc4d","Type":"ContainerStarted","Data":"83834bc4f50b3cc9c8a639430c0409fbd9397ddd829fe2c49ad2c07798a015a7"}
Sep 17 13:59:23.547914 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:23.547549    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0327e6ab-d63a-438a-b925-fc60d270bc4d","Type":"ContainerStarted","Data":"e91a74b213acc6bcab1ad65dd1c7715edf396b5dc45288cb911193875b0d8276"}
Sep 17 13:59:23.547914 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:23.547589    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0327e6ab-d63a-438a-b925-fc60d270bc4d","Type":"ContainerStarted","Data":"eb5427aa0aa955fe5f17526bc784953f9cd06f2bb2950ba26881754a3de0d6f3"}
Sep 17 13:59:23.568165 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:23.568137    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a29762f9-e0d5-43cd-b19f-2db732145b53" path="/var/lib/kubelet/pods/a29762f9-e0d5-43cd-b19f-2db732145b53/volumes"
Sep 17 13:59:23.587098 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:23.587046    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.572005134 podStartE2EDuration="2.587031951s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.026817ms" isStatefulPod=false podCreationTimestamp="2026-09-17 13:59:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 13:59:23.58605732 +0000 UTC m=+232.528200969" watchObservedRunningTime="2026-09-17 13:59:23.587031951 +0000 UTC m=+232.529175596"
Sep 17 13:59:24.059326 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:24.059291    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-56cd85c7b4-4cj6v"]
Sep 17 13:59:24.063455 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:24.063437    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-56cd85c7b4-4cj6v"
Sep 17 13:59:24.067542 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:24.067520    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 17 13:59:24.067542 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:24.067521    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 17 13:59:24.067912 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:24.067893    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 17 13:59:24.069180 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:24.069158    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 17 13:59:24.069752 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:24.069731    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 17 13:59:24.070100 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:24.070086    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-fxvmz\""
Sep 17 13:59:24.077892 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:24.077870    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 17 13:59:24.081080 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:24.081059    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-56cd85c7b4-4cj6v"]
Sep 17 13:59:24.131825 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:24.131800    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/90da0389-baf3-4223-8975-ff83b7d02a71-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-56cd85c7b4-4cj6v\" (UID: \"90da0389-baf3-4223-8975-ff83b7d02a71\") " pod="openshift-monitoring/telemeter-client-56cd85c7b4-4cj6v"
Sep 17 13:59:24.131908 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:24.131831    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90da0389-baf3-4223-8975-ff83b7d02a71-metrics-client-ca\") pod \"telemeter-client-56cd85c7b4-4cj6v\" (UID: \"90da0389-baf3-4223-8975-ff83b7d02a71\") " pod="openshift-monitoring/telemeter-client-56cd85c7b4-4cj6v"
Sep 17 13:59:24.131908 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:24.131849    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/90da0389-baf3-4223-8975-ff83b7d02a71-federate-client-tls\") pod \"telemeter-client-56cd85c7b4-4cj6v\" (UID: \"90da0389-baf3-4223-8975-ff83b7d02a71\") " pod="openshift-monitoring/telemeter-client-56cd85c7b4-4cj6v"
Sep 17 13:59:24.131908 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:24.131877    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/90da0389-baf3-4223-8975-ff83b7d02a71-telemeter-client-tls\") pod \"telemeter-client-56cd85c7b4-4cj6v\" (UID: \"90da0389-baf3-4223-8975-ff83b7d02a71\") " pod="openshift-monitoring/telemeter-client-56cd85c7b4-4cj6v"
Sep 17 13:59:24.132031 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:24.131937    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/90da0389-baf3-4223-8975-ff83b7d02a71-secret-telemeter-client\") pod \"telemeter-client-56cd85c7b4-4cj6v\" (UID: \"90da0389-baf3-4223-8975-ff83b7d02a71\") " pod="openshift-monitoring/telemeter-client-56cd85c7b4-4cj6v"
Sep 17 13:59:24.132031 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:24.131973    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jv94\" (UniqueName: \"kubernetes.io/projected/90da0389-baf3-4223-8975-ff83b7d02a71-kube-api-access-8jv94\") pod \"telemeter-client-56cd85c7b4-4cj6v\" (UID: \"90da0389-baf3-4223-8975-ff83b7d02a71\") " pod="openshift-monitoring/telemeter-client-56cd85c7b4-4cj6v"
Sep 17 13:59:24.132031 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:24.132013    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90da0389-baf3-4223-8975-ff83b7d02a71-serving-certs-ca-bundle\") pod \"telemeter-client-56cd85c7b4-4cj6v\" (UID: \"90da0389-baf3-4223-8975-ff83b7d02a71\") " pod="openshift-monitoring/telemeter-client-56cd85c7b4-4cj6v"
Sep 17 13:59:24.132031 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:24.132028    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90da0389-baf3-4223-8975-ff83b7d02a71-telemeter-trusted-ca-bundle\") pod \"telemeter-client-56cd85c7b4-4cj6v\" (UID: \"90da0389-baf3-4223-8975-ff83b7d02a71\") " pod="openshift-monitoring/telemeter-client-56cd85c7b4-4cj6v"
Sep 17 13:59:24.232800 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:24.232764    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90da0389-baf3-4223-8975-ff83b7d02a71-serving-certs-ca-bundle\") pod \"telemeter-client-56cd85c7b4-4cj6v\" (UID: \"90da0389-baf3-4223-8975-ff83b7d02a71\") " pod="openshift-monitoring/telemeter-client-56cd85c7b4-4cj6v"
Sep 17 13:59:24.232951 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:24.232825    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90da0389-baf3-4223-8975-ff83b7d02a71-telemeter-trusted-ca-bundle\") pod \"telemeter-client-56cd85c7b4-4cj6v\" (UID: \"90da0389-baf3-4223-8975-ff83b7d02a71\") " pod="openshift-monitoring/telemeter-client-56cd85c7b4-4cj6v"
Sep 17 13:59:24.232951 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:24.232866    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/90da0389-baf3-4223-8975-ff83b7d02a71-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-56cd85c7b4-4cj6v\" (UID: \"90da0389-baf3-4223-8975-ff83b7d02a71\") " pod="openshift-monitoring/telemeter-client-56cd85c7b4-4cj6v"
Sep 17 13:59:24.232951 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:24.232910    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90da0389-baf3-4223-8975-ff83b7d02a71-metrics-client-ca\") pod \"telemeter-client-56cd85c7b4-4cj6v\" (UID: \"90da0389-baf3-4223-8975-ff83b7d02a71\") " pod="openshift-monitoring/telemeter-client-56cd85c7b4-4cj6v"
Sep 17 13:59:24.232951 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:24.232941    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/90da0389-baf3-4223-8975-ff83b7d02a71-federate-client-tls\") pod \"telemeter-client-56cd85c7b4-4cj6v\" (UID: \"90da0389-baf3-4223-8975-ff83b7d02a71\") " pod="openshift-monitoring/telemeter-client-56cd85c7b4-4cj6v"
Sep 17 13:59:24.233747 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:24.233453    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/90da0389-baf3-4223-8975-ff83b7d02a71-telemeter-client-tls\") pod \"telemeter-client-56cd85c7b4-4cj6v\" (UID: \"90da0389-baf3-4223-8975-ff83b7d02a71\") " pod="openshift-monitoring/telemeter-client-56cd85c7b4-4cj6v"
Sep 17 13:59:24.233747 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:24.233574    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/90da0389-baf3-4223-8975-ff83b7d02a71-secret-telemeter-client\") pod \"telemeter-client-56cd85c7b4-4cj6v\" (UID: \"90da0389-baf3-4223-8975-ff83b7d02a71\") " pod="openshift-monitoring/telemeter-client-56cd85c7b4-4cj6v"
Sep 17 13:59:24.233747 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:24.233610    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jv94\" (UniqueName: \"kubernetes.io/projected/90da0389-baf3-4223-8975-ff83b7d02a71-kube-api-access-8jv94\") pod \"telemeter-client-56cd85c7b4-4cj6v\" (UID: \"90da0389-baf3-4223-8975-ff83b7d02a71\") " pod="openshift-monitoring/telemeter-client-56cd85c7b4-4cj6v"
Sep 17 13:59:24.233747 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:24.233709    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90da0389-baf3-4223-8975-ff83b7d02a71-serving-certs-ca-bundle\") pod \"telemeter-client-56cd85c7b4-4cj6v\" (UID: \"90da0389-baf3-4223-8975-ff83b7d02a71\") " pod="openshift-monitoring/telemeter-client-56cd85c7b4-4cj6v"
Sep 17 13:59:24.235340 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:24.234852    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90da0389-baf3-4223-8975-ff83b7d02a71-telemeter-trusted-ca-bundle\") pod \"telemeter-client-56cd85c7b4-4cj6v\" (UID: \"90da0389-baf3-4223-8975-ff83b7d02a71\") " pod="openshift-monitoring/telemeter-client-56cd85c7b4-4cj6v"
Sep 17 13:59:24.235340 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:24.235039    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90da0389-baf3-4223-8975-ff83b7d02a71-metrics-client-ca\") pod \"telemeter-client-56cd85c7b4-4cj6v\" (UID: \"90da0389-baf3-4223-8975-ff83b7d02a71\") " pod="openshift-monitoring/telemeter-client-56cd85c7b4-4cj6v"
Sep 17 13:59:24.236408 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:24.236346    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/90da0389-baf3-4223-8975-ff83b7d02a71-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-56cd85c7b4-4cj6v\" (UID: \"90da0389-baf3-4223-8975-ff83b7d02a71\") " pod="openshift-monitoring/telemeter-client-56cd85c7b4-4cj6v"
Sep 17 13:59:24.237902 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:24.237875    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/90da0389-baf3-4223-8975-ff83b7d02a71-telemeter-client-tls\") pod \"telemeter-client-56cd85c7b4-4cj6v\" (UID: \"90da0389-baf3-4223-8975-ff83b7d02a71\") " pod="openshift-monitoring/telemeter-client-56cd85c7b4-4cj6v"
Sep 17 13:59:24.240612 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:24.240590    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/90da0389-baf3-4223-8975-ff83b7d02a71-federate-client-tls\") pod \"telemeter-client-56cd85c7b4-4cj6v\" (UID: \"90da0389-baf3-4223-8975-ff83b7d02a71\") " pod="openshift-monitoring/telemeter-client-56cd85c7b4-4cj6v"
Sep 17 13:59:24.243633 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:24.243612    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/90da0389-baf3-4223-8975-ff83b7d02a71-secret-telemeter-client\") pod \"telemeter-client-56cd85c7b4-4cj6v\" (UID: \"90da0389-baf3-4223-8975-ff83b7d02a71\") " pod="openshift-monitoring/telemeter-client-56cd85c7b4-4cj6v"
Sep 17 13:59:24.244333 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:24.244309    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jv94\" (UniqueName: \"kubernetes.io/projected/90da0389-baf3-4223-8975-ff83b7d02a71-kube-api-access-8jv94\") pod \"telemeter-client-56cd85c7b4-4cj6v\" (UID: \"90da0389-baf3-4223-8975-ff83b7d02a71\") " pod="openshift-monitoring/telemeter-client-56cd85c7b4-4cj6v"
Sep 17 13:59:24.375640 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:24.375564    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-56cd85c7b4-4cj6v"
Sep 17 13:59:24.659322 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:24.659244    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-56cd85c7b4-4cj6v"]
Sep 17 13:59:24.663306 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:59:24.663278    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90da0389_baf3_4223_8975_ff83b7d02a71.slice/crio-d00e77c3418bc6beee0d87c9a3719d158765aac09a18cd922a8e61033585d25b WatchSource:0}: Error finding container d00e77c3418bc6beee0d87c9a3719d158765aac09a18cd922a8e61033585d25b: Status 404 returned error can't find the container with id d00e77c3418bc6beee0d87c9a3719d158765aac09a18cd922a8e61033585d25b
Sep 17 13:59:25.557924 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:25.557882    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-56cd85c7b4-4cj6v" event={"ID":"90da0389-baf3-4223-8975-ff83b7d02a71","Type":"ContainerStarted","Data":"d00e77c3418bc6beee0d87c9a3719d158765aac09a18cd922a8e61033585d25b"}
Sep 17 13:59:26.563992 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:26.563927    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-56cd85c7b4-4cj6v" event={"ID":"90da0389-baf3-4223-8975-ff83b7d02a71","Type":"ContainerStarted","Data":"312e05dbd4dc7b5d62044f340e8785b41258b9242c5170fc27318799d7787acc"}
Sep 17 13:59:26.563992 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:26.563988    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-56cd85c7b4-4cj6v" event={"ID":"90da0389-baf3-4223-8975-ff83b7d02a71","Type":"ContainerStarted","Data":"701f8fc9a75c7d26264826bda748b5c12fbabc4d714a6940e5815cad68b397f7"}
Sep 17 13:59:26.564424 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:26.564019    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-56cd85c7b4-4cj6v" event={"ID":"90da0389-baf3-4223-8975-ff83b7d02a71","Type":"ContainerStarted","Data":"c181eb8ed351757c1441855aea0bcfaeaf6965abecc64dbe2be197c5c7add0a1"}
Sep 17 13:59:26.592557 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:26.592516    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-56cd85c7b4-4cj6v" podStartSLOduration=0.943434144 podStartE2EDuration="2.592502433s" totalImagesPullingTime="1.649068289s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 13:59:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 13:59:26.591787353 +0000 UTC m=+235.533931010" watchObservedRunningTime="2026-09-17 13:59:26.592502433 +0000 UTC m=+235.534646078"
Sep 17 13:59:26.775238 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:26.775204    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d96ff4668-hl7ht"]
Sep 17 13:59:26.778545 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:26.778527    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d96ff4668-hl7ht"
Sep 17 13:59:26.793538 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:26.793507    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d96ff4668-hl7ht"]
Sep 17 13:59:26.857292 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:26.857239    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/25d1972a-3f16-411d-a498-5b0ccbf5b07a-console-serving-cert\") pod \"console-5d96ff4668-hl7ht\" (UID: \"25d1972a-3f16-411d-a498-5b0ccbf5b07a\") " pod="openshift-console/console-5d96ff4668-hl7ht"
Sep 17 13:59:26.857292 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:26.857265    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/25d1972a-3f16-411d-a498-5b0ccbf5b07a-service-ca\") pod \"console-5d96ff4668-hl7ht\" (UID: \"25d1972a-3f16-411d-a498-5b0ccbf5b07a\") " pod="openshift-console/console-5d96ff4668-hl7ht"
Sep 17 13:59:26.857292 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:26.857284    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/25d1972a-3f16-411d-a498-5b0ccbf5b07a-console-oauth-config\") pod \"console-5d96ff4668-hl7ht\" (UID: \"25d1972a-3f16-411d-a498-5b0ccbf5b07a\") " pod="openshift-console/console-5d96ff4668-hl7ht"
Sep 17 13:59:26.857497 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:26.857308    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25d1972a-3f16-411d-a498-5b0ccbf5b07a-trusted-ca-bundle\") pod \"console-5d96ff4668-hl7ht\" (UID: \"25d1972a-3f16-411d-a498-5b0ccbf5b07a\") " pod="openshift-console/console-5d96ff4668-hl7ht"
Sep 17 13:59:26.857497 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:26.857325    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsrkw\" (UniqueName: \"kubernetes.io/projected/25d1972a-3f16-411d-a498-5b0ccbf5b07a-kube-api-access-hsrkw\") pod \"console-5d96ff4668-hl7ht\" (UID: \"25d1972a-3f16-411d-a498-5b0ccbf5b07a\") " pod="openshift-console/console-5d96ff4668-hl7ht"
Sep 17 13:59:26.857497 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:26.857386    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/25d1972a-3f16-411d-a498-5b0ccbf5b07a-oauth-serving-cert\") pod \"console-5d96ff4668-hl7ht\" (UID: \"25d1972a-3f16-411d-a498-5b0ccbf5b07a\") " pod="openshift-console/console-5d96ff4668-hl7ht"
Sep 17 13:59:26.857497 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:26.857447    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/25d1972a-3f16-411d-a498-5b0ccbf5b07a-console-config\") pod \"console-5d96ff4668-hl7ht\" (UID: \"25d1972a-3f16-411d-a498-5b0ccbf5b07a\") " pod="openshift-console/console-5d96ff4668-hl7ht"
Sep 17 13:59:26.858759 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:26.858733    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:59:26.958168 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:26.958145    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/25d1972a-3f16-411d-a498-5b0ccbf5b07a-console-serving-cert\") pod \"console-5d96ff4668-hl7ht\" (UID: \"25d1972a-3f16-411d-a498-5b0ccbf5b07a\") " pod="openshift-console/console-5d96ff4668-hl7ht"
Sep 17 13:59:26.958255 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:26.958171    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/25d1972a-3f16-411d-a498-5b0ccbf5b07a-service-ca\") pod \"console-5d96ff4668-hl7ht\" (UID: \"25d1972a-3f16-411d-a498-5b0ccbf5b07a\") " pod="openshift-console/console-5d96ff4668-hl7ht"
Sep 17 13:59:26.958255 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:26.958189    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/25d1972a-3f16-411d-a498-5b0ccbf5b07a-console-oauth-config\") pod \"console-5d96ff4668-hl7ht\" (UID: \"25d1972a-3f16-411d-a498-5b0ccbf5b07a\") " pod="openshift-console/console-5d96ff4668-hl7ht"
Sep 17 13:59:26.958255 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:26.958212    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25d1972a-3f16-411d-a498-5b0ccbf5b07a-trusted-ca-bundle\") pod \"console-5d96ff4668-hl7ht\" (UID: \"25d1972a-3f16-411d-a498-5b0ccbf5b07a\") " pod="openshift-console/console-5d96ff4668-hl7ht"
Sep 17 13:59:26.958255 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:26.958236    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsrkw\" (UniqueName: \"kubernetes.io/projected/25d1972a-3f16-411d-a498-5b0ccbf5b07a-kube-api-access-hsrkw\") pod \"console-5d96ff4668-hl7ht\" (UID: \"25d1972a-3f16-411d-a498-5b0ccbf5b07a\") " pod="openshift-console/console-5d96ff4668-hl7ht"
Sep 17 13:59:26.958416 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:26.958264    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/25d1972a-3f16-411d-a498-5b0ccbf5b07a-oauth-serving-cert\") pod \"console-5d96ff4668-hl7ht\" (UID: \"25d1972a-3f16-411d-a498-5b0ccbf5b07a\") " pod="openshift-console/console-5d96ff4668-hl7ht"
Sep 17 13:59:26.958416 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:26.958302    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/25d1972a-3f16-411d-a498-5b0ccbf5b07a-console-config\") pod \"console-5d96ff4668-hl7ht\" (UID: \"25d1972a-3f16-411d-a498-5b0ccbf5b07a\") " pod="openshift-console/console-5d96ff4668-hl7ht"
Sep 17 13:59:26.958964 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:26.958940    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/25d1972a-3f16-411d-a498-5b0ccbf5b07a-service-ca\") pod \"console-5d96ff4668-hl7ht\" (UID: \"25d1972a-3f16-411d-a498-5b0ccbf5b07a\") " pod="openshift-console/console-5d96ff4668-hl7ht"
Sep 17 13:59:26.959087 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:26.959063    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/25d1972a-3f16-411d-a498-5b0ccbf5b07a-console-config\") pod \"console-5d96ff4668-hl7ht\" (UID: \"25d1972a-3f16-411d-a498-5b0ccbf5b07a\") " pod="openshift-console/console-5d96ff4668-hl7ht"
Sep 17 13:59:26.959149 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:26.959119    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/25d1972a-3f16-411d-a498-5b0ccbf5b07a-oauth-serving-cert\") pod \"console-5d96ff4668-hl7ht\" (UID: \"25d1972a-3f16-411d-a498-5b0ccbf5b07a\") " pod="openshift-console/console-5d96ff4668-hl7ht"
Sep 17 13:59:26.959277 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:26.959261    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25d1972a-3f16-411d-a498-5b0ccbf5b07a-trusted-ca-bundle\") pod \"console-5d96ff4668-hl7ht\" (UID: \"25d1972a-3f16-411d-a498-5b0ccbf5b07a\") " pod="openshift-console/console-5d96ff4668-hl7ht"
Sep 17 13:59:26.961144 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:26.961121    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/25d1972a-3f16-411d-a498-5b0ccbf5b07a-console-serving-cert\") pod \"console-5d96ff4668-hl7ht\" (UID: \"25d1972a-3f16-411d-a498-5b0ccbf5b07a\") " pod="openshift-console/console-5d96ff4668-hl7ht"
Sep 17 13:59:26.961227 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:26.961208    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/25d1972a-3f16-411d-a498-5b0ccbf5b07a-console-oauth-config\") pod \"console-5d96ff4668-hl7ht\" (UID: \"25d1972a-3f16-411d-a498-5b0ccbf5b07a\") " pod="openshift-console/console-5d96ff4668-hl7ht"
Sep 17 13:59:26.962502 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:26.962477    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:59:26.967806 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:26.967786    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsrkw\" (UniqueName: \"kubernetes.io/projected/25d1972a-3f16-411d-a498-5b0ccbf5b07a-kube-api-access-hsrkw\") pod \"console-5d96ff4668-hl7ht\" (UID: \"25d1972a-3f16-411d-a498-5b0ccbf5b07a\") " pod="openshift-console/console-5d96ff4668-hl7ht"
Sep 17 13:59:27.087467 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:27.087433    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d96ff4668-hl7ht"
Sep 17 13:59:27.370184 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:27.370156    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d96ff4668-hl7ht"]
Sep 17 13:59:27.373012 ip-10-0-129-158 kubenswrapper[2560]: W0917 13:59:27.372980    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25d1972a_3f16_411d_a498_5b0ccbf5b07a.slice/crio-7370dcaac67891a7a401f73a675fa5aa340ffdd3c73dcb0cb47d4e32799b06e5 WatchSource:0}: Error finding container 7370dcaac67891a7a401f73a675fa5aa340ffdd3c73dcb0cb47d4e32799b06e5: Status 404 returned error can't find the container with id 7370dcaac67891a7a401f73a675fa5aa340ffdd3c73dcb0cb47d4e32799b06e5
Sep 17 13:59:27.569714 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:27.569685    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d96ff4668-hl7ht" event={"ID":"25d1972a-3f16-411d-a498-5b0ccbf5b07a","Type":"ContainerStarted","Data":"fac991e297bf13827b27b6d3322d31127042a6e73ae10bc6fab19c430da5f087"}
Sep 17 13:59:27.570186 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:27.569735    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d96ff4668-hl7ht" event={"ID":"25d1972a-3f16-411d-a498-5b0ccbf5b07a","Type":"ContainerStarted","Data":"7370dcaac67891a7a401f73a675fa5aa340ffdd3c73dcb0cb47d4e32799b06e5"}
Sep 17 13:59:27.591918 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:27.591871    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5d96ff4668-hl7ht" podStartSLOduration=1.5918556910000001 podStartE2EDuration="1.591855691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 13:59:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 13:59:27.590826214 +0000 UTC m=+236.532969861" watchObservedRunningTime="2026-09-17 13:59:27.591855691 +0000 UTC m=+236.533999336"
Sep 17 13:59:37.088479 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:37.088429    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d96ff4668-hl7ht"
Sep 17 13:59:37.088933 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:37.088517    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5d96ff4668-hl7ht"
Sep 17 13:59:37.093099 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:37.093072    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5d96ff4668-hl7ht"
Sep 17 13:59:37.612270 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:37.612245    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5d96ff4668-hl7ht"
Sep 17 13:59:37.667479 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:37.667429    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-688858bc96-kqg68"]
Sep 17 13:59:37.802650 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:37.802617    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:59:37.903583 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:37.903509    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:59:37.974024 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:37.973993    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 13:59:43.896322 ip-10-0-129-158 kubenswrapper[2560]: I0917 13:59:43.896284    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:00:01.678630 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:01.678589    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:00:01.795595 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:01.795559    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:00:02.694732 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:02.694694    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-688858bc96-kqg68" podUID="46cd612c-7614-48d7-823f-115a955f2856" containerName="console" containerID="cri-o://8fbfb6375f3a19264dc26598ef2c4204e0130387203e48c3f8014673b0bf57d8" gracePeriod=15
Sep 17 14:00:02.931696 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:02.931673    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-688858bc96-kqg68"
Sep 17 14:00:03.027442 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:03.027359    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/46cd612c-7614-48d7-823f-115a955f2856-service-ca\") pod \"46cd612c-7614-48d7-823f-115a955f2856\" (UID: \"46cd612c-7614-48d7-823f-115a955f2856\") "
Sep 17 14:00:03.027442 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:03.027404    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/46cd612c-7614-48d7-823f-115a955f2856-console-serving-cert\") pod \"46cd612c-7614-48d7-823f-115a955f2856\" (UID: \"46cd612c-7614-48d7-823f-115a955f2856\") "
Sep 17 14:00:03.027442 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:03.027433    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mszpq\" (UniqueName: \"kubernetes.io/projected/46cd612c-7614-48d7-823f-115a955f2856-kube-api-access-mszpq\") pod \"46cd612c-7614-48d7-823f-115a955f2856\" (UID: \"46cd612c-7614-48d7-823f-115a955f2856\") "
Sep 17 14:00:03.027719 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:03.027491    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/46cd612c-7614-48d7-823f-115a955f2856-oauth-serving-cert\") pod \"46cd612c-7614-48d7-823f-115a955f2856\" (UID: \"46cd612c-7614-48d7-823f-115a955f2856\") "
Sep 17 14:00:03.027719 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:03.027536    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/46cd612c-7614-48d7-823f-115a955f2856-console-oauth-config\") pod \"46cd612c-7614-48d7-823f-115a955f2856\" (UID: \"46cd612c-7614-48d7-823f-115a955f2856\") "
Sep 17 14:00:03.027719 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:03.027553    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/46cd612c-7614-48d7-823f-115a955f2856-console-config\") pod \"46cd612c-7614-48d7-823f-115a955f2856\" (UID: \"46cd612c-7614-48d7-823f-115a955f2856\") "
Sep 17 14:00:03.027719 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:03.027594    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46cd612c-7614-48d7-823f-115a955f2856-trusted-ca-bundle\") pod \"46cd612c-7614-48d7-823f-115a955f2856\" (UID: \"46cd612c-7614-48d7-823f-115a955f2856\") "
Sep 17 14:00:03.027990 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:03.027936    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46cd612c-7614-48d7-823f-115a955f2856-service-ca" (OuterVolumeSpecName: "service-ca") pod "46cd612c-7614-48d7-823f-115a955f2856" (UID: "46cd612c-7614-48d7-823f-115a955f2856"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:00:03.028111 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:03.028068    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46cd612c-7614-48d7-823f-115a955f2856-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "46cd612c-7614-48d7-823f-115a955f2856" (UID: "46cd612c-7614-48d7-823f-115a955f2856"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:00:03.028111 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:03.028074    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46cd612c-7614-48d7-823f-115a955f2856-console-config" (OuterVolumeSpecName: "console-config") pod "46cd612c-7614-48d7-823f-115a955f2856" (UID: "46cd612c-7614-48d7-823f-115a955f2856"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:00:03.028245 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:03.028108    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46cd612c-7614-48d7-823f-115a955f2856-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "46cd612c-7614-48d7-823f-115a955f2856" (UID: "46cd612c-7614-48d7-823f-115a955f2856"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:00:03.029651 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:03.029618    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46cd612c-7614-48d7-823f-115a955f2856-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "46cd612c-7614-48d7-823f-115a955f2856" (UID: "46cd612c-7614-48d7-823f-115a955f2856"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:00:03.029741 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:03.029659    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46cd612c-7614-48d7-823f-115a955f2856-kube-api-access-mszpq" (OuterVolumeSpecName: "kube-api-access-mszpq") pod "46cd612c-7614-48d7-823f-115a955f2856" (UID: "46cd612c-7614-48d7-823f-115a955f2856"). InnerVolumeSpecName "kube-api-access-mszpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:00:03.029741 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:03.029665    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46cd612c-7614-48d7-823f-115a955f2856-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "46cd612c-7614-48d7-823f-115a955f2856" (UID: "46cd612c-7614-48d7-823f-115a955f2856"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:00:03.128293 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:03.128252    2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/46cd612c-7614-48d7-823f-115a955f2856-service-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:00:03.128440 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:03.128315    2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/46cd612c-7614-48d7-823f-115a955f2856-console-serving-cert\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:00:03.128440 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:03.128331    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mszpq\" (UniqueName: \"kubernetes.io/projected/46cd612c-7614-48d7-823f-115a955f2856-kube-api-access-mszpq\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:00:03.128440 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:03.128344    2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/46cd612c-7614-48d7-823f-115a955f2856-oauth-serving-cert\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:00:03.128440 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:03.128355    2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/46cd612c-7614-48d7-823f-115a955f2856-console-oauth-config\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:00:03.128440 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:03.128369    2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/46cd612c-7614-48d7-823f-115a955f2856-console-config\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:00:03.128440 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:03.128380    2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46cd612c-7614-48d7-823f-115a955f2856-trusted-ca-bundle\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:00:03.711975 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:03.711936    2560 generic.go:356] "Generic (PLEG): container finished" podID="46cd612c-7614-48d7-823f-115a955f2856" containerID="8fbfb6375f3a19264dc26598ef2c4204e0130387203e48c3f8014673b0bf57d8" exitCode=0
Sep 17 14:00:03.712419 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:03.711985    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-688858bc96-kqg68"
Sep 17 14:00:03.712419 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:03.712016    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-688858bc96-kqg68" event={"ID":"46cd612c-7614-48d7-823f-115a955f2856","Type":"ContainerDied","Data":"8fbfb6375f3a19264dc26598ef2c4204e0130387203e48c3f8014673b0bf57d8"}
Sep 17 14:00:03.712419 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:03.712065    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-688858bc96-kqg68" event={"ID":"46cd612c-7614-48d7-823f-115a955f2856","Type":"ContainerDied","Data":"7c01531bc2320e8efafe2c1de29ee0ca224bf525a1c81f39f6c0d590cc72f882"}
Sep 17 14:00:03.712419 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:03.712085    2560 scope.go:117] "RemoveContainer" containerID="8fbfb6375f3a19264dc26598ef2c4204e0130387203e48c3f8014673b0bf57d8"
Sep 17 14:00:03.720026 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:03.720004    2560 scope.go:117] "RemoveContainer" containerID="8fbfb6375f3a19264dc26598ef2c4204e0130387203e48c3f8014673b0bf57d8"
Sep 17 14:00:03.720311 ip-10-0-129-158 kubenswrapper[2560]: E0917 14:00:03.720292    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fbfb6375f3a19264dc26598ef2c4204e0130387203e48c3f8014673b0bf57d8\": container with ID starting with 8fbfb6375f3a19264dc26598ef2c4204e0130387203e48c3f8014673b0bf57d8 not found: ID does not exist" containerID="8fbfb6375f3a19264dc26598ef2c4204e0130387203e48c3f8014673b0bf57d8"
Sep 17 14:00:03.720361 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:03.720319    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fbfb6375f3a19264dc26598ef2c4204e0130387203e48c3f8014673b0bf57d8"} err="failed to get container status \"8fbfb6375f3a19264dc26598ef2c4204e0130387203e48c3f8014673b0bf57d8\": rpc error: code = NotFound desc = could not find container \"8fbfb6375f3a19264dc26598ef2c4204e0130387203e48c3f8014673b0bf57d8\": container with ID starting with 8fbfb6375f3a19264dc26598ef2c4204e0130387203e48c3f8014673b0bf57d8 not found: ID does not exist"
Sep 17 14:00:03.731317 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:03.731298    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-688858bc96-kqg68"]
Sep 17 14:00:03.734346 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:03.734325    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-688858bc96-kqg68"]
Sep 17 14:00:05.568499 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:05.568448    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46cd612c-7614-48d7-823f-115a955f2856" path="/var/lib/kubelet/pods/46cd612c-7614-48d7-823f-115a955f2856/volumes"
Sep 17 14:00:14.545388 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:14.545354    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:00:31.498676 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:31.498651    2560 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 14:00:34.270906 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:34.270871    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:00:34.377389 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:34.377351    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:00:34.393662 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:34.393636    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6b4b86fd88-vbkb4"]
Sep 17 14:00:34.393968 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:34.393941    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46cd612c-7614-48d7-823f-115a955f2856" containerName="console"
Sep 17 14:00:34.393968 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:34.393957    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="46cd612c-7614-48d7-823f-115a955f2856" containerName="console"
Sep 17 14:00:34.394137 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:34.394062    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="46cd612c-7614-48d7-823f-115a955f2856" containerName="console"
Sep 17 14:00:34.396948 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:34.396930    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b4b86fd88-vbkb4"
Sep 17 14:00:34.408178 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:34.408155    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b4b86fd88-vbkb4"]
Sep 17 14:00:34.444175 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:34.444151    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4fe051b3-4179-4b6d-9360-da9257afae32-console-serving-cert\") pod \"console-6b4b86fd88-vbkb4\" (UID: \"4fe051b3-4179-4b6d-9360-da9257afae32\") " pod="openshift-console/console-6b4b86fd88-vbkb4"
Sep 17 14:00:34.444293 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:34.444183    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4fe051b3-4179-4b6d-9360-da9257afae32-console-oauth-config\") pod \"console-6b4b86fd88-vbkb4\" (UID: \"4fe051b3-4179-4b6d-9360-da9257afae32\") " pod="openshift-console/console-6b4b86fd88-vbkb4"
Sep 17 14:00:34.444293 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:34.444211    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpb6v\" (UniqueName: \"kubernetes.io/projected/4fe051b3-4179-4b6d-9360-da9257afae32-kube-api-access-bpb6v\") pod \"console-6b4b86fd88-vbkb4\" (UID: \"4fe051b3-4179-4b6d-9360-da9257afae32\") " pod="openshift-console/console-6b4b86fd88-vbkb4"
Sep 17 14:00:34.444293 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:34.444248    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4fe051b3-4179-4b6d-9360-da9257afae32-service-ca\") pod \"console-6b4b86fd88-vbkb4\" (UID: \"4fe051b3-4179-4b6d-9360-da9257afae32\") " pod="openshift-console/console-6b4b86fd88-vbkb4"
Sep 17 14:00:34.444398 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:34.444291    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4fe051b3-4179-4b6d-9360-da9257afae32-console-config\") pod \"console-6b4b86fd88-vbkb4\" (UID: \"4fe051b3-4179-4b6d-9360-da9257afae32\") " pod="openshift-console/console-6b4b86fd88-vbkb4"
Sep 17 14:00:34.444398 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:34.444359    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4fe051b3-4179-4b6d-9360-da9257afae32-trusted-ca-bundle\") pod \"console-6b4b86fd88-vbkb4\" (UID: \"4fe051b3-4179-4b6d-9360-da9257afae32\") " pod="openshift-console/console-6b4b86fd88-vbkb4"
Sep 17 14:00:34.444481 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:34.444401    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4fe051b3-4179-4b6d-9360-da9257afae32-oauth-serving-cert\") pod \"console-6b4b86fd88-vbkb4\" (UID: \"4fe051b3-4179-4b6d-9360-da9257afae32\") " pod="openshift-console/console-6b4b86fd88-vbkb4"
Sep 17 14:00:34.486812 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:34.486786    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:00:34.545402 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:34.545356    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4fe051b3-4179-4b6d-9360-da9257afae32-service-ca\") pod \"console-6b4b86fd88-vbkb4\" (UID: \"4fe051b3-4179-4b6d-9360-da9257afae32\") " pod="openshift-console/console-6b4b86fd88-vbkb4"
Sep 17 14:00:34.545402 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:34.545385    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4fe051b3-4179-4b6d-9360-da9257afae32-console-config\") pod \"console-6b4b86fd88-vbkb4\" (UID: \"4fe051b3-4179-4b6d-9360-da9257afae32\") " pod="openshift-console/console-6b4b86fd88-vbkb4"
Sep 17 14:00:34.545582 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:34.545414    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4fe051b3-4179-4b6d-9360-da9257afae32-trusted-ca-bundle\") pod \"console-6b4b86fd88-vbkb4\" (UID: \"4fe051b3-4179-4b6d-9360-da9257afae32\") " pod="openshift-console/console-6b4b86fd88-vbkb4"
Sep 17 14:00:34.545582 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:34.545455    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4fe051b3-4179-4b6d-9360-da9257afae32-oauth-serving-cert\") pod \"console-6b4b86fd88-vbkb4\" (UID: \"4fe051b3-4179-4b6d-9360-da9257afae32\") " pod="openshift-console/console-6b4b86fd88-vbkb4"
Sep 17 14:00:34.545582 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:34.545501    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4fe051b3-4179-4b6d-9360-da9257afae32-console-serving-cert\") pod \"console-6b4b86fd88-vbkb4\" (UID: \"4fe051b3-4179-4b6d-9360-da9257afae32\") " pod="openshift-console/console-6b4b86fd88-vbkb4"
Sep 17 14:00:34.545582 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:34.545518    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4fe051b3-4179-4b6d-9360-da9257afae32-console-oauth-config\") pod \"console-6b4b86fd88-vbkb4\" (UID: \"4fe051b3-4179-4b6d-9360-da9257afae32\") " pod="openshift-console/console-6b4b86fd88-vbkb4"
Sep 17 14:00:34.545582 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:34.545533    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpb6v\" (UniqueName: \"kubernetes.io/projected/4fe051b3-4179-4b6d-9360-da9257afae32-kube-api-access-bpb6v\") pod \"console-6b4b86fd88-vbkb4\" (UID: \"4fe051b3-4179-4b6d-9360-da9257afae32\") " pod="openshift-console/console-6b4b86fd88-vbkb4"
Sep 17 14:00:34.546153 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:34.546133    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4fe051b3-4179-4b6d-9360-da9257afae32-service-ca\") pod \"console-6b4b86fd88-vbkb4\" (UID: \"4fe051b3-4179-4b6d-9360-da9257afae32\") " pod="openshift-console/console-6b4b86fd88-vbkb4"
Sep 17 14:00:34.546290 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:34.546266    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4fe051b3-4179-4b6d-9360-da9257afae32-console-config\") pod \"console-6b4b86fd88-vbkb4\" (UID: \"4fe051b3-4179-4b6d-9360-da9257afae32\") " pod="openshift-console/console-6b4b86fd88-vbkb4"
Sep 17 14:00:34.546368 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:34.546318    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4fe051b3-4179-4b6d-9360-da9257afae32-trusted-ca-bundle\") pod \"console-6b4b86fd88-vbkb4\" (UID: \"4fe051b3-4179-4b6d-9360-da9257afae32\") " pod="openshift-console/console-6b4b86fd88-vbkb4"
Sep 17 14:00:34.546368 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:34.546323    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4fe051b3-4179-4b6d-9360-da9257afae32-oauth-serving-cert\") pod \"console-6b4b86fd88-vbkb4\" (UID: \"4fe051b3-4179-4b6d-9360-da9257afae32\") " pod="openshift-console/console-6b4b86fd88-vbkb4"
Sep 17 14:00:34.547910 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:34.547889    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4fe051b3-4179-4b6d-9360-da9257afae32-console-oauth-config\") pod \"console-6b4b86fd88-vbkb4\" (UID: \"4fe051b3-4179-4b6d-9360-da9257afae32\") " pod="openshift-console/console-6b4b86fd88-vbkb4"
Sep 17 14:00:34.547984 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:34.547950    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4fe051b3-4179-4b6d-9360-da9257afae32-console-serving-cert\") pod \"console-6b4b86fd88-vbkb4\" (UID: \"4fe051b3-4179-4b6d-9360-da9257afae32\") " pod="openshift-console/console-6b4b86fd88-vbkb4"
Sep 17 14:00:34.554363 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:34.554343    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpb6v\" (UniqueName: \"kubernetes.io/projected/4fe051b3-4179-4b6d-9360-da9257afae32-kube-api-access-bpb6v\") pod \"console-6b4b86fd88-vbkb4\" (UID: \"4fe051b3-4179-4b6d-9360-da9257afae32\") " pod="openshift-console/console-6b4b86fd88-vbkb4"
Sep 17 14:00:34.705962 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:34.705935    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b4b86fd88-vbkb4"
Sep 17 14:00:34.978034 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:34.978014    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b4b86fd88-vbkb4"]
Sep 17 14:00:34.980036 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:00:34.980010    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fe051b3_4179_4b6d_9360_da9257afae32.slice/crio-b53a8515b401e4ee49da8e88c1bb47c325c5bc41762a2a08b4d0b0fafd075948 WatchSource:0}: Error finding container b53a8515b401e4ee49da8e88c1bb47c325c5bc41762a2a08b4d0b0fafd075948: Status 404 returned error can't find the container with id b53a8515b401e4ee49da8e88c1bb47c325c5bc41762a2a08b4d0b0fafd075948
Sep 17 14:00:35.847827 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:35.847796    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b4b86fd88-vbkb4" event={"ID":"4fe051b3-4179-4b6d-9360-da9257afae32","Type":"ContainerStarted","Data":"4de72121a738070060e9ec6028e7b3efe422d3308dbd26b328ef813078cbe4d8"}
Sep 17 14:00:35.848179 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:35.847841    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b4b86fd88-vbkb4" event={"ID":"4fe051b3-4179-4b6d-9360-da9257afae32","Type":"ContainerStarted","Data":"b53a8515b401e4ee49da8e88c1bb47c325c5bc41762a2a08b4d0b0fafd075948"}
Sep 17 14:00:35.868009 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:35.867964    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6b4b86fd88-vbkb4" podStartSLOduration=1.867949425 podStartE2EDuration="1.867949425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:00:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:00:35.867159213 +0000 UTC m=+304.809302857" watchObservedRunningTime="2026-09-17 14:00:35.867949425 +0000 UTC m=+304.810093070"
Sep 17 14:00:44.706182 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:44.706144    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6b4b86fd88-vbkb4"
Sep 17 14:00:44.706692 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:44.706231    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6b4b86fd88-vbkb4"
Sep 17 14:00:44.710955 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:44.710934    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6b4b86fd88-vbkb4"
Sep 17 14:00:44.888386 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:44.888360    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6b4b86fd88-vbkb4"
Sep 17 14:00:44.948861 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:44.948832    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d96ff4668-hl7ht"]
Sep 17 14:00:45.076542 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:45.076508    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:00:45.180159 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:45.180120    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:00:45.252613 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:45.252563    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:00:46.211444 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:46.211409    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:00:46.282609 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:46.282577    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:00:55.607491 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:55.607436    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:00:55.709552 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:55.709520    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:00:56.382507 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:00:56.382440    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:01:09.968071 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:01:09.968031    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5d96ff4668-hl7ht" podUID="25d1972a-3f16-411d-a498-5b0ccbf5b07a" containerName="console" containerID="cri-o://fac991e297bf13827b27b6d3322d31127042a6e73ae10bc6fab19c430da5f087" gracePeriod=15
Sep 17 14:01:10.201687 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:01:10.201666    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d96ff4668-hl7ht"
Sep 17 14:01:10.330586 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:01:10.330562    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/25d1972a-3f16-411d-a498-5b0ccbf5b07a-console-serving-cert\") pod \"25d1972a-3f16-411d-a498-5b0ccbf5b07a\" (UID: \"25d1972a-3f16-411d-a498-5b0ccbf5b07a\") "
Sep 17 14:01:10.330731 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:01:10.330599    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25d1972a-3f16-411d-a498-5b0ccbf5b07a-trusted-ca-bundle\") pod \"25d1972a-3f16-411d-a498-5b0ccbf5b07a\" (UID: \"25d1972a-3f16-411d-a498-5b0ccbf5b07a\") "
Sep 17 14:01:10.330731 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:01:10.330626    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/25d1972a-3f16-411d-a498-5b0ccbf5b07a-oauth-serving-cert\") pod \"25d1972a-3f16-411d-a498-5b0ccbf5b07a\" (UID: \"25d1972a-3f16-411d-a498-5b0ccbf5b07a\") "
Sep 17 14:01:10.330731 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:01:10.330649    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/25d1972a-3f16-411d-a498-5b0ccbf5b07a-console-config\") pod \"25d1972a-3f16-411d-a498-5b0ccbf5b07a\" (UID: \"25d1972a-3f16-411d-a498-5b0ccbf5b07a\") "
Sep 17 14:01:10.330893 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:01:10.330766    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/25d1972a-3f16-411d-a498-5b0ccbf5b07a-console-oauth-config\") pod \"25d1972a-3f16-411d-a498-5b0ccbf5b07a\" (UID: \"25d1972a-3f16-411d-a498-5b0ccbf5b07a\") "
Sep 17 14:01:10.330893 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:01:10.330808    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hsrkw\" (UniqueName: \"kubernetes.io/projected/25d1972a-3f16-411d-a498-5b0ccbf5b07a-kube-api-access-hsrkw\") pod \"25d1972a-3f16-411d-a498-5b0ccbf5b07a\" (UID: \"25d1972a-3f16-411d-a498-5b0ccbf5b07a\") "
Sep 17 14:01:10.330893 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:01:10.330836    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/25d1972a-3f16-411d-a498-5b0ccbf5b07a-service-ca\") pod \"25d1972a-3f16-411d-a498-5b0ccbf5b07a\" (UID: \"25d1972a-3f16-411d-a498-5b0ccbf5b07a\") "
Sep 17 14:01:10.331048 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:01:10.331011    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25d1972a-3f16-411d-a498-5b0ccbf5b07a-console-config" (OuterVolumeSpecName: "console-config") pod "25d1972a-3f16-411d-a498-5b0ccbf5b07a" (UID: "25d1972a-3f16-411d-a498-5b0ccbf5b07a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:01:10.331048 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:01:10.331029    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25d1972a-3f16-411d-a498-5b0ccbf5b07a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "25d1972a-3f16-411d-a498-5b0ccbf5b07a" (UID: "25d1972a-3f16-411d-a498-5b0ccbf5b07a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:01:10.331136 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:01:10.331053    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25d1972a-3f16-411d-a498-5b0ccbf5b07a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "25d1972a-3f16-411d-a498-5b0ccbf5b07a" (UID: "25d1972a-3f16-411d-a498-5b0ccbf5b07a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:01:10.331136 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:01:10.331120    2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25d1972a-3f16-411d-a498-5b0ccbf5b07a-trusted-ca-bundle\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:01:10.331136 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:01:10.331134    2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/25d1972a-3f16-411d-a498-5b0ccbf5b07a-oauth-serving-cert\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:01:10.331270 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:01:10.331143    2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/25d1972a-3f16-411d-a498-5b0ccbf5b07a-console-config\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:01:10.331358 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:01:10.331335    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25d1972a-3f16-411d-a498-5b0ccbf5b07a-service-ca" (OuterVolumeSpecName: "service-ca") pod "25d1972a-3f16-411d-a498-5b0ccbf5b07a" (UID: "25d1972a-3f16-411d-a498-5b0ccbf5b07a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:01:10.332776 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:01:10.332754    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25d1972a-3f16-411d-a498-5b0ccbf5b07a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "25d1972a-3f16-411d-a498-5b0ccbf5b07a" (UID: "25d1972a-3f16-411d-a498-5b0ccbf5b07a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:01:10.332776 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:01:10.332765    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25d1972a-3f16-411d-a498-5b0ccbf5b07a-kube-api-access-hsrkw" (OuterVolumeSpecName: "kube-api-access-hsrkw") pod "25d1972a-3f16-411d-a498-5b0ccbf5b07a" (UID: "25d1972a-3f16-411d-a498-5b0ccbf5b07a"). InnerVolumeSpecName "kube-api-access-hsrkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:01:10.332925 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:01:10.332774    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25d1972a-3f16-411d-a498-5b0ccbf5b07a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "25d1972a-3f16-411d-a498-5b0ccbf5b07a" (UID: "25d1972a-3f16-411d-a498-5b0ccbf5b07a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:01:10.432423 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:01:10.432388    2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/25d1972a-3f16-411d-a498-5b0ccbf5b07a-console-serving-cert\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:01:10.432423 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:01:10.432421    2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/25d1972a-3f16-411d-a498-5b0ccbf5b07a-console-oauth-config\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:01:10.432423 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:01:10.432430    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsrkw\" (UniqueName: \"kubernetes.io/projected/25d1972a-3f16-411d-a498-5b0ccbf5b07a-kube-api-access-hsrkw\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:01:10.432608 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:01:10.432439    2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/25d1972a-3f16-411d-a498-5b0ccbf5b07a-service-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:01:10.985784 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:01:10.985705    2560 generic.go:356] "Generic (PLEG): container finished" podID="25d1972a-3f16-411d-a498-5b0ccbf5b07a" containerID="fac991e297bf13827b27b6d3322d31127042a6e73ae10bc6fab19c430da5f087" exitCode=0
Sep 17 14:01:10.985784 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:01:10.985748    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d96ff4668-hl7ht"
Sep 17 14:01:10.985784 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:01:10.985769    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d96ff4668-hl7ht" event={"ID":"25d1972a-3f16-411d-a498-5b0ccbf5b07a","Type":"ContainerDied","Data":"fac991e297bf13827b27b6d3322d31127042a6e73ae10bc6fab19c430da5f087"}
Sep 17 14:01:10.986224 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:01:10.985808    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d96ff4668-hl7ht" event={"ID":"25d1972a-3f16-411d-a498-5b0ccbf5b07a","Type":"ContainerDied","Data":"7370dcaac67891a7a401f73a675fa5aa340ffdd3c73dcb0cb47d4e32799b06e5"}
Sep 17 14:01:10.986224 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:01:10.985827    2560 scope.go:117] "RemoveContainer" containerID="fac991e297bf13827b27b6d3322d31127042a6e73ae10bc6fab19c430da5f087"
Sep 17 14:01:10.994237 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:01:10.994220    2560 scope.go:117] "RemoveContainer" containerID="fac991e297bf13827b27b6d3322d31127042a6e73ae10bc6fab19c430da5f087"
Sep 17 14:01:10.994502 ip-10-0-129-158 kubenswrapper[2560]: E0917 14:01:10.994482    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fac991e297bf13827b27b6d3322d31127042a6e73ae10bc6fab19c430da5f087\": container with ID starting with fac991e297bf13827b27b6d3322d31127042a6e73ae10bc6fab19c430da5f087 not found: ID does not exist" containerID="fac991e297bf13827b27b6d3322d31127042a6e73ae10bc6fab19c430da5f087"
Sep 17 14:01:10.994571 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:01:10.994513    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fac991e297bf13827b27b6d3322d31127042a6e73ae10bc6fab19c430da5f087"} err="failed to get container status \"fac991e297bf13827b27b6d3322d31127042a6e73ae10bc6fab19c430da5f087\": rpc error: code = NotFound desc = could not find container \"fac991e297bf13827b27b6d3322d31127042a6e73ae10bc6fab19c430da5f087\": container with ID starting with fac991e297bf13827b27b6d3322d31127042a6e73ae10bc6fab19c430da5f087 not found: ID does not exist"
Sep 17 14:01:11.006141 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:01:11.006119    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d96ff4668-hl7ht"]
Sep 17 14:01:11.010676 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:01:11.010657    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5d96ff4668-hl7ht"]
Sep 17 14:01:11.568244 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:01:11.568213    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25d1972a-3f16-411d-a498-5b0ccbf5b07a" path="/var/lib/kubelet/pods/25d1972a-3f16-411d-a498-5b0ccbf5b07a/volumes"
Sep 17 14:01:17.639811 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:01:17.639774    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:02:09.907373 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:02:09.907339    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:02:11.595358 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:02:11.595324    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:02:39.532387 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:02:39.532351    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:03:01.873375 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:01.873342    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5bbb6b5f44-9pjsb"]
Sep 17 14:03:01.873922 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:01.873814    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25d1972a-3f16-411d-a498-5b0ccbf5b07a" containerName="console"
Sep 17 14:03:01.873922 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:01.873834    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="25d1972a-3f16-411d-a498-5b0ccbf5b07a" containerName="console"
Sep 17 14:03:01.873993 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:01.873933    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="25d1972a-3f16-411d-a498-5b0ccbf5b07a" containerName="console"
Sep 17 14:03:01.876856 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:01.876836    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bbb6b5f44-9pjsb"
Sep 17 14:03:01.899730 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:01.899708    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bbb6b5f44-9pjsb"]
Sep 17 14:03:01.934824 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:01.934795    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:03:02.006281 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:02.006256    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3514d6f7-c94a-4c0a-9dfc-e15cad52cf68-console-serving-cert\") pod \"console-5bbb6b5f44-9pjsb\" (UID: \"3514d6f7-c94a-4c0a-9dfc-e15cad52cf68\") " pod="openshift-console/console-5bbb6b5f44-9pjsb"
Sep 17 14:03:02.006395 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:02.006296    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3514d6f7-c94a-4c0a-9dfc-e15cad52cf68-service-ca\") pod \"console-5bbb6b5f44-9pjsb\" (UID: \"3514d6f7-c94a-4c0a-9dfc-e15cad52cf68\") " pod="openshift-console/console-5bbb6b5f44-9pjsb"
Sep 17 14:03:02.006395 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:02.006322    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3514d6f7-c94a-4c0a-9dfc-e15cad52cf68-console-config\") pod \"console-5bbb6b5f44-9pjsb\" (UID: \"3514d6f7-c94a-4c0a-9dfc-e15cad52cf68\") " pod="openshift-console/console-5bbb6b5f44-9pjsb"
Sep 17 14:03:02.006543 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:02.006395    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z799v\" (UniqueName: \"kubernetes.io/projected/3514d6f7-c94a-4c0a-9dfc-e15cad52cf68-kube-api-access-z799v\") pod \"console-5bbb6b5f44-9pjsb\" (UID: \"3514d6f7-c94a-4c0a-9dfc-e15cad52cf68\") " pod="openshift-console/console-5bbb6b5f44-9pjsb"
Sep 17 14:03:02.006543 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:02.006436    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3514d6f7-c94a-4c0a-9dfc-e15cad52cf68-trusted-ca-bundle\") pod \"console-5bbb6b5f44-9pjsb\" (UID: \"3514d6f7-c94a-4c0a-9dfc-e15cad52cf68\") " pod="openshift-console/console-5bbb6b5f44-9pjsb"
Sep 17 14:03:02.006543 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:02.006487    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3514d6f7-c94a-4c0a-9dfc-e15cad52cf68-oauth-serving-cert\") pod \"console-5bbb6b5f44-9pjsb\" (UID: \"3514d6f7-c94a-4c0a-9dfc-e15cad52cf68\") " pod="openshift-console/console-5bbb6b5f44-9pjsb"
Sep 17 14:03:02.006543 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:02.006529    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3514d6f7-c94a-4c0a-9dfc-e15cad52cf68-console-oauth-config\") pod \"console-5bbb6b5f44-9pjsb\" (UID: \"3514d6f7-c94a-4c0a-9dfc-e15cad52cf68\") " pod="openshift-console/console-5bbb6b5f44-9pjsb"
Sep 17 14:03:02.012172 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:02.012152    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:03:02.107425 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:02.107396    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3514d6f7-c94a-4c0a-9dfc-e15cad52cf68-console-config\") pod \"console-5bbb6b5f44-9pjsb\" (UID: \"3514d6f7-c94a-4c0a-9dfc-e15cad52cf68\") " pod="openshift-console/console-5bbb6b5f44-9pjsb"
Sep 17 14:03:02.107571 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:02.107434    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z799v\" (UniqueName: \"kubernetes.io/projected/3514d6f7-c94a-4c0a-9dfc-e15cad52cf68-kube-api-access-z799v\") pod \"console-5bbb6b5f44-9pjsb\" (UID: \"3514d6f7-c94a-4c0a-9dfc-e15cad52cf68\") " pod="openshift-console/console-5bbb6b5f44-9pjsb"
Sep 17 14:03:02.107571 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:02.107456    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3514d6f7-c94a-4c0a-9dfc-e15cad52cf68-trusted-ca-bundle\") pod \"console-5bbb6b5f44-9pjsb\" (UID: \"3514d6f7-c94a-4c0a-9dfc-e15cad52cf68\") " pod="openshift-console/console-5bbb6b5f44-9pjsb"
Sep 17 14:03:02.107571 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:02.107493    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3514d6f7-c94a-4c0a-9dfc-e15cad52cf68-oauth-serving-cert\") pod \"console-5bbb6b5f44-9pjsb\" (UID: \"3514d6f7-c94a-4c0a-9dfc-e15cad52cf68\") " pod="openshift-console/console-5bbb6b5f44-9pjsb"
Sep 17 14:03:02.107571 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:02.107519    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3514d6f7-c94a-4c0a-9dfc-e15cad52cf68-console-oauth-config\") pod \"console-5bbb6b5f44-9pjsb\" (UID: \"3514d6f7-c94a-4c0a-9dfc-e15cad52cf68\") " pod="openshift-console/console-5bbb6b5f44-9pjsb"
Sep 17 14:03:02.107571 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:02.107536    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3514d6f7-c94a-4c0a-9dfc-e15cad52cf68-console-serving-cert\") pod \"console-5bbb6b5f44-9pjsb\" (UID: \"3514d6f7-c94a-4c0a-9dfc-e15cad52cf68\") " pod="openshift-console/console-5bbb6b5f44-9pjsb"
Sep 17 14:03:02.107571 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:02.107569    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3514d6f7-c94a-4c0a-9dfc-e15cad52cf68-service-ca\") pod \"console-5bbb6b5f44-9pjsb\" (UID: \"3514d6f7-c94a-4c0a-9dfc-e15cad52cf68\") " pod="openshift-console/console-5bbb6b5f44-9pjsb"
Sep 17 14:03:02.108250 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:02.108226    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3514d6f7-c94a-4c0a-9dfc-e15cad52cf68-console-config\") pod \"console-5bbb6b5f44-9pjsb\" (UID: \"3514d6f7-c94a-4c0a-9dfc-e15cad52cf68\") " pod="openshift-console/console-5bbb6b5f44-9pjsb"
Sep 17 14:03:02.108391 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:02.108367    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3514d6f7-c94a-4c0a-9dfc-e15cad52cf68-trusted-ca-bundle\") pod \"console-5bbb6b5f44-9pjsb\" (UID: \"3514d6f7-c94a-4c0a-9dfc-e15cad52cf68\") " pod="openshift-console/console-5bbb6b5f44-9pjsb"
Sep 17 14:03:02.108391 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:02.108343    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3514d6f7-c94a-4c0a-9dfc-e15cad52cf68-oauth-serving-cert\") pod \"console-5bbb6b5f44-9pjsb\" (UID: \"3514d6f7-c94a-4c0a-9dfc-e15cad52cf68\") " pod="openshift-console/console-5bbb6b5f44-9pjsb"
Sep 17 14:03:02.108391 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:02.108290    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3514d6f7-c94a-4c0a-9dfc-e15cad52cf68-service-ca\") pod \"console-5bbb6b5f44-9pjsb\" (UID: \"3514d6f7-c94a-4c0a-9dfc-e15cad52cf68\") " pod="openshift-console/console-5bbb6b5f44-9pjsb"
Sep 17 14:03:02.109932 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:02.109904    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3514d6f7-c94a-4c0a-9dfc-e15cad52cf68-console-serving-cert\") pod \"console-5bbb6b5f44-9pjsb\" (UID: \"3514d6f7-c94a-4c0a-9dfc-e15cad52cf68\") " pod="openshift-console/console-5bbb6b5f44-9pjsb"
Sep 17 14:03:02.110019 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:02.109943    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3514d6f7-c94a-4c0a-9dfc-e15cad52cf68-console-oauth-config\") pod \"console-5bbb6b5f44-9pjsb\" (UID: \"3514d6f7-c94a-4c0a-9dfc-e15cad52cf68\") " pod="openshift-console/console-5bbb6b5f44-9pjsb"
Sep 17 14:03:02.116324 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:02.116292    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z799v\" (UniqueName: \"kubernetes.io/projected/3514d6f7-c94a-4c0a-9dfc-e15cad52cf68-kube-api-access-z799v\") pod \"console-5bbb6b5f44-9pjsb\" (UID: \"3514d6f7-c94a-4c0a-9dfc-e15cad52cf68\") " pod="openshift-console/console-5bbb6b5f44-9pjsb"
Sep 17 14:03:02.186439 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:02.186377    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bbb6b5f44-9pjsb"
Sep 17 14:03:02.460129 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:02.460047    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bbb6b5f44-9pjsb"]
Sep 17 14:03:02.467402 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:03:02.467374    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3514d6f7_c94a_4c0a_9dfc_e15cad52cf68.slice/crio-fb17fa567c213ec8f18efdde8676e246a8d756220f9e9e11422defef5ba53a87 WatchSource:0}: Error finding container fb17fa567c213ec8f18efdde8676e246a8d756220f9e9e11422defef5ba53a87: Status 404 returned error can't find the container with id fb17fa567c213ec8f18efdde8676e246a8d756220f9e9e11422defef5ba53a87
Sep 17 14:03:03.419482 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:03.419434    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bbb6b5f44-9pjsb" event={"ID":"3514d6f7-c94a-4c0a-9dfc-e15cad52cf68","Type":"ContainerStarted","Data":"0a79b6f33e99043747f604e32963f950f56d7b3f77e76ff1f7a243d18f3de6f4"}
Sep 17 14:03:03.419908 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:03.419498    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bbb6b5f44-9pjsb" event={"ID":"3514d6f7-c94a-4c0a-9dfc-e15cad52cf68","Type":"ContainerStarted","Data":"fb17fa567c213ec8f18efdde8676e246a8d756220f9e9e11422defef5ba53a87"}
Sep 17 14:03:03.448148 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:03.448100    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5bbb6b5f44-9pjsb" podStartSLOduration=2.448086466 podStartE2EDuration="2.448086466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:03:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:03:03.445916673 +0000 UTC m=+452.388060330" watchObservedRunningTime="2026-09-17 14:03:03.448086466 +0000 UTC m=+452.390230177"
Sep 17 14:03:12.186725 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:12.186625    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5bbb6b5f44-9pjsb"
Sep 17 14:03:12.186725 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:12.186697    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5bbb6b5f44-9pjsb"
Sep 17 14:03:12.191191 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:12.191170    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5bbb6b5f44-9pjsb"
Sep 17 14:03:12.457877 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:12.457796    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5bbb6b5f44-9pjsb"
Sep 17 14:03:12.520189 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:12.520156    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b4b86fd88-vbkb4"]
Sep 17 14:03:12.657431 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:12.657395    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:03:12.770895 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:12.770821    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:03:26.903431 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:26.903396    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407pdlg"]
Sep 17 14:03:26.907696 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:26.907677    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407pdlg"
Sep 17 14:03:26.910073 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:26.910052    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 14:03:26.910193 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:26.910125    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 14:03:26.910193 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:26.910141    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-d62p6\""
Sep 17 14:03:26.917670 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:26.917647    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407pdlg"]
Sep 17 14:03:27.009652 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:27.009617    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/27f919c4-ad28-4dd2-9a7d-0baae581e785-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407pdlg\" (UID: \"27f919c4-ad28-4dd2-9a7d-0baae581e785\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407pdlg"
Sep 17 14:03:27.009817 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:27.009663    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-687wp\" (UniqueName: \"kubernetes.io/projected/27f919c4-ad28-4dd2-9a7d-0baae581e785-kube-api-access-687wp\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407pdlg\" (UID: \"27f919c4-ad28-4dd2-9a7d-0baae581e785\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407pdlg"
Sep 17 14:03:27.009817 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:27.009692    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/27f919c4-ad28-4dd2-9a7d-0baae581e785-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407pdlg\" (UID: \"27f919c4-ad28-4dd2-9a7d-0baae581e785\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407pdlg"
Sep 17 14:03:27.110970 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:27.110923    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/27f919c4-ad28-4dd2-9a7d-0baae581e785-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407pdlg\" (UID: \"27f919c4-ad28-4dd2-9a7d-0baae581e785\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407pdlg"
Sep 17 14:03:27.111105 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:27.110993    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-687wp\" (UniqueName: \"kubernetes.io/projected/27f919c4-ad28-4dd2-9a7d-0baae581e785-kube-api-access-687wp\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407pdlg\" (UID: \"27f919c4-ad28-4dd2-9a7d-0baae581e785\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407pdlg"
Sep 17 14:03:27.111105 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:27.111027    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/27f919c4-ad28-4dd2-9a7d-0baae581e785-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407pdlg\" (UID: \"27f919c4-ad28-4dd2-9a7d-0baae581e785\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407pdlg"
Sep 17 14:03:27.111350 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:27.111334    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/27f919c4-ad28-4dd2-9a7d-0baae581e785-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407pdlg\" (UID: \"27f919c4-ad28-4dd2-9a7d-0baae581e785\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407pdlg"
Sep 17 14:03:27.111411 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:27.111395    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/27f919c4-ad28-4dd2-9a7d-0baae581e785-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407pdlg\" (UID: \"27f919c4-ad28-4dd2-9a7d-0baae581e785\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407pdlg"
Sep 17 14:03:27.119425 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:27.119395    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-687wp\" (UniqueName: \"kubernetes.io/projected/27f919c4-ad28-4dd2-9a7d-0baae581e785-kube-api-access-687wp\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407pdlg\" (UID: \"27f919c4-ad28-4dd2-9a7d-0baae581e785\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407pdlg"
Sep 17 14:03:27.216930 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:27.216844    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407pdlg"
Sep 17 14:03:27.487972 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:27.487849    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407pdlg"]
Sep 17 14:03:27.490673 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:03:27.490642    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27f919c4_ad28_4dd2_9a7d_0baae581e785.slice/crio-6e661c74b87f6cfeb5deadf11950c63d4a58929b16aa7e9a2c69f8ee336b0e6e WatchSource:0}: Error finding container 6e661c74b87f6cfeb5deadf11950c63d4a58929b16aa7e9a2c69f8ee336b0e6e: Status 404 returned error can't find the container with id 6e661c74b87f6cfeb5deadf11950c63d4a58929b16aa7e9a2c69f8ee336b0e6e
Sep 17 14:03:27.492619 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:27.492604    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 14:03:27.514898 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:27.514861    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407pdlg" event={"ID":"27f919c4-ad28-4dd2-9a7d-0baae581e785","Type":"ContainerStarted","Data":"6e661c74b87f6cfeb5deadf11950c63d4a58929b16aa7e9a2c69f8ee336b0e6e"}
Sep 17 14:03:33.545345 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:33.545266    2560 generic.go:356] "Generic (PLEG): container finished" podID="27f919c4-ad28-4dd2-9a7d-0baae581e785" containerID="99fb20affac571ad55e4099d8f70c6cbda16e8d9863fce028fa2bb87ecef4bdc" exitCode=0
Sep 17 14:03:33.545345 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:33.545332    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407pdlg" event={"ID":"27f919c4-ad28-4dd2-9a7d-0baae581e785","Type":"ContainerDied","Data":"99fb20affac571ad55e4099d8f70c6cbda16e8d9863fce028fa2bb87ecef4bdc"}
Sep 17 14:03:36.558537 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:36.558502    2560 generic.go:356] "Generic (PLEG): container finished" podID="27f919c4-ad28-4dd2-9a7d-0baae581e785" containerID="ca3643662c42ca87a08c30f0ac5a3b7daf260b403bbeaa87c3bd1c1000ab65d4" exitCode=0
Sep 17 14:03:36.558915 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:36.558569    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407pdlg" event={"ID":"27f919c4-ad28-4dd2-9a7d-0baae581e785","Type":"ContainerDied","Data":"ca3643662c42ca87a08c30f0ac5a3b7daf260b403bbeaa87c3bd1c1000ab65d4"}
Sep 17 14:03:37.542797 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:37.542736    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6b4b86fd88-vbkb4" podUID="4fe051b3-4179-4b6d-9360-da9257afae32" containerName="console" containerID="cri-o://4de72121a738070060e9ec6028e7b3efe422d3308dbd26b328ef813078cbe4d8" gracePeriod=15
Sep 17 14:03:37.794645 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:37.794587    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b4b86fd88-vbkb4"
Sep 17 14:03:37.795007 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:37.794869    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4fe051b3-4179-4b6d-9360-da9257afae32-service-ca\") pod \"4fe051b3-4179-4b6d-9360-da9257afae32\" (UID: \"4fe051b3-4179-4b6d-9360-da9257afae32\") "
Sep 17 14:03:37.795074 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:37.795007    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4fe051b3-4179-4b6d-9360-da9257afae32-console-oauth-config\") pod \"4fe051b3-4179-4b6d-9360-da9257afae32\" (UID: \"4fe051b3-4179-4b6d-9360-da9257afae32\") "
Sep 17 14:03:37.795074 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:37.795064    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4fe051b3-4179-4b6d-9360-da9257afae32-console-config\") pod \"4fe051b3-4179-4b6d-9360-da9257afae32\" (UID: \"4fe051b3-4179-4b6d-9360-da9257afae32\") "
Sep 17 14:03:37.795182 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:37.795090    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4fe051b3-4179-4b6d-9360-da9257afae32-trusted-ca-bundle\") pod \"4fe051b3-4179-4b6d-9360-da9257afae32\" (UID: \"4fe051b3-4179-4b6d-9360-da9257afae32\") "
Sep 17 14:03:37.795182 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:37.795123    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4fe051b3-4179-4b6d-9360-da9257afae32-oauth-serving-cert\") pod \"4fe051b3-4179-4b6d-9360-da9257afae32\" (UID: \"4fe051b3-4179-4b6d-9360-da9257afae32\") "
Sep 17 14:03:37.795182 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:37.795166    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bpb6v\" (UniqueName: \"kubernetes.io/projected/4fe051b3-4179-4b6d-9360-da9257afae32-kube-api-access-bpb6v\") pod \"4fe051b3-4179-4b6d-9360-da9257afae32\" (UID: \"4fe051b3-4179-4b6d-9360-da9257afae32\") "
Sep 17 14:03:37.795325 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:37.795192    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4fe051b3-4179-4b6d-9360-da9257afae32-console-serving-cert\") pod \"4fe051b3-4179-4b6d-9360-da9257afae32\" (UID: \"4fe051b3-4179-4b6d-9360-da9257afae32\") "
Sep 17 14:03:37.795325 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:37.795233    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fe051b3-4179-4b6d-9360-da9257afae32-service-ca" (OuterVolumeSpecName: "service-ca") pod "4fe051b3-4179-4b6d-9360-da9257afae32" (UID: "4fe051b3-4179-4b6d-9360-da9257afae32"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:03:37.795476 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:37.795448    2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4fe051b3-4179-4b6d-9360-da9257afae32-service-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:03:37.795741 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:37.795703    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fe051b3-4179-4b6d-9360-da9257afae32-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4fe051b3-4179-4b6d-9360-da9257afae32" (UID: "4fe051b3-4179-4b6d-9360-da9257afae32"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:03:37.795741 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:37.795728    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fe051b3-4179-4b6d-9360-da9257afae32-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "4fe051b3-4179-4b6d-9360-da9257afae32" (UID: "4fe051b3-4179-4b6d-9360-da9257afae32"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:03:37.795923 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:37.795766    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fe051b3-4179-4b6d-9360-da9257afae32-console-config" (OuterVolumeSpecName: "console-config") pod "4fe051b3-4179-4b6d-9360-da9257afae32" (UID: "4fe051b3-4179-4b6d-9360-da9257afae32"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:03:37.797559 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:37.797435    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fe051b3-4179-4b6d-9360-da9257afae32-kube-api-access-bpb6v" (OuterVolumeSpecName: "kube-api-access-bpb6v") pod "4fe051b3-4179-4b6d-9360-da9257afae32" (UID: "4fe051b3-4179-4b6d-9360-da9257afae32"). InnerVolumeSpecName "kube-api-access-bpb6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:03:37.797559 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:37.797484    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fe051b3-4179-4b6d-9360-da9257afae32-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4fe051b3-4179-4b6d-9360-da9257afae32" (UID: "4fe051b3-4179-4b6d-9360-da9257afae32"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:03:37.797719 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:37.797699    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fe051b3-4179-4b6d-9360-da9257afae32-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4fe051b3-4179-4b6d-9360-da9257afae32" (UID: "4fe051b3-4179-4b6d-9360-da9257afae32"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:03:37.896095 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:37.896063    2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4fe051b3-4179-4b6d-9360-da9257afae32-console-config\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:03:37.896095 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:37.896098    2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4fe051b3-4179-4b6d-9360-da9257afae32-trusted-ca-bundle\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:03:37.896288 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:37.896111    2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4fe051b3-4179-4b6d-9360-da9257afae32-oauth-serving-cert\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:03:37.896288 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:37.896119    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpb6v\" (UniqueName: \"kubernetes.io/projected/4fe051b3-4179-4b6d-9360-da9257afae32-kube-api-access-bpb6v\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:03:37.896288 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:37.896129    2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4fe051b3-4179-4b6d-9360-da9257afae32-console-serving-cert\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:03:37.896288 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:37.896137    2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4fe051b3-4179-4b6d-9360-da9257afae32-console-oauth-config\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:03:38.570991 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:38.570955    2560 generic.go:356] "Generic (PLEG): container finished" podID="4fe051b3-4179-4b6d-9360-da9257afae32" containerID="4de72121a738070060e9ec6028e7b3efe422d3308dbd26b328ef813078cbe4d8" exitCode=0
Sep 17 14:03:38.571189 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:38.570996    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b4b86fd88-vbkb4"
Sep 17 14:03:38.571189 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:38.571012    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b4b86fd88-vbkb4" event={"ID":"4fe051b3-4179-4b6d-9360-da9257afae32","Type":"ContainerDied","Data":"4de72121a738070060e9ec6028e7b3efe422d3308dbd26b328ef813078cbe4d8"}
Sep 17 14:03:38.571189 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:38.571048    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b4b86fd88-vbkb4" event={"ID":"4fe051b3-4179-4b6d-9360-da9257afae32","Type":"ContainerDied","Data":"b53a8515b401e4ee49da8e88c1bb47c325c5bc41762a2a08b4d0b0fafd075948"}
Sep 17 14:03:38.571189 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:38.571070    2560 scope.go:117] "RemoveContainer" containerID="4de72121a738070060e9ec6028e7b3efe422d3308dbd26b328ef813078cbe4d8"
Sep 17 14:03:38.580825 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:38.580807    2560 scope.go:117] "RemoveContainer" containerID="4de72121a738070060e9ec6028e7b3efe422d3308dbd26b328ef813078cbe4d8"
Sep 17 14:03:38.581104 ip-10-0-129-158 kubenswrapper[2560]: E0917 14:03:38.581083    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4de72121a738070060e9ec6028e7b3efe422d3308dbd26b328ef813078cbe4d8\": container with ID starting with 4de72121a738070060e9ec6028e7b3efe422d3308dbd26b328ef813078cbe4d8 not found: ID does not exist" containerID="4de72121a738070060e9ec6028e7b3efe422d3308dbd26b328ef813078cbe4d8"
Sep 17 14:03:38.581175 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:38.581114    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4de72121a738070060e9ec6028e7b3efe422d3308dbd26b328ef813078cbe4d8"} err="failed to get container status \"4de72121a738070060e9ec6028e7b3efe422d3308dbd26b328ef813078cbe4d8\": rpc error: code = NotFound desc = could not find container \"4de72121a738070060e9ec6028e7b3efe422d3308dbd26b328ef813078cbe4d8\": container with ID starting with 4de72121a738070060e9ec6028e7b3efe422d3308dbd26b328ef813078cbe4d8 not found: ID does not exist"
Sep 17 14:03:38.593248 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:38.593220    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b4b86fd88-vbkb4"]
Sep 17 14:03:38.598896 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:38.598876    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6b4b86fd88-vbkb4"]
Sep 17 14:03:39.570561 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:39.570529    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fe051b3-4179-4b6d-9360-da9257afae32" path="/var/lib/kubelet/pods/4fe051b3-4179-4b6d-9360-da9257afae32/volumes"
Sep 17 14:03:43.596121 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:43.596086    2560 generic.go:356] "Generic (PLEG): container finished" podID="27f919c4-ad28-4dd2-9a7d-0baae581e785" containerID="2d3e837a477c12a6f512fc33b1d2ba0dd7ad44a7260105eea02e13a18c2f5002" exitCode=0
Sep 17 14:03:43.596517 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:43.596159    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407pdlg" event={"ID":"27f919c4-ad28-4dd2-9a7d-0baae581e785","Type":"ContainerDied","Data":"2d3e837a477c12a6f512fc33b1d2ba0dd7ad44a7260105eea02e13a18c2f5002"}
Sep 17 14:03:44.720202 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:44.720180    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407pdlg"
Sep 17 14:03:44.747930 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:44.747907    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/27f919c4-ad28-4dd2-9a7d-0baae581e785-bundle\") pod \"27f919c4-ad28-4dd2-9a7d-0baae581e785\" (UID: \"27f919c4-ad28-4dd2-9a7d-0baae581e785\") "
Sep 17 14:03:44.748045 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:44.747950    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/27f919c4-ad28-4dd2-9a7d-0baae581e785-util\") pod \"27f919c4-ad28-4dd2-9a7d-0baae581e785\" (UID: \"27f919c4-ad28-4dd2-9a7d-0baae581e785\") "
Sep 17 14:03:44.748045 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:44.747985    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-687wp\" (UniqueName: \"kubernetes.io/projected/27f919c4-ad28-4dd2-9a7d-0baae581e785-kube-api-access-687wp\") pod \"27f919c4-ad28-4dd2-9a7d-0baae581e785\" (UID: \"27f919c4-ad28-4dd2-9a7d-0baae581e785\") "
Sep 17 14:03:44.748720 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:44.748692    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f919c4-ad28-4dd2-9a7d-0baae581e785-bundle" (OuterVolumeSpecName: "bundle") pod "27f919c4-ad28-4dd2-9a7d-0baae581e785" (UID: "27f919c4-ad28-4dd2-9a7d-0baae581e785"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:03:44.750154 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:44.750125    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27f919c4-ad28-4dd2-9a7d-0baae581e785-kube-api-access-687wp" (OuterVolumeSpecName: "kube-api-access-687wp") pod "27f919c4-ad28-4dd2-9a7d-0baae581e785" (UID: "27f919c4-ad28-4dd2-9a7d-0baae581e785"). InnerVolumeSpecName "kube-api-access-687wp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:03:44.752700 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:44.752677    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f919c4-ad28-4dd2-9a7d-0baae581e785-util" (OuterVolumeSpecName: "util") pod "27f919c4-ad28-4dd2-9a7d-0baae581e785" (UID: "27f919c4-ad28-4dd2-9a7d-0baae581e785"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:03:44.849360 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:44.849335    2560 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/27f919c4-ad28-4dd2-9a7d-0baae581e785-util\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:03:44.849360 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:44.849356    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-687wp\" (UniqueName: \"kubernetes.io/projected/27f919c4-ad28-4dd2-9a7d-0baae581e785-kube-api-access-687wp\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:03:44.849509 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:44.849366    2560 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/27f919c4-ad28-4dd2-9a7d-0baae581e785-bundle\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:03:45.606180 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:45.606150    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407pdlg" event={"ID":"27f919c4-ad28-4dd2-9a7d-0baae581e785","Type":"ContainerDied","Data":"6e661c74b87f6cfeb5deadf11950c63d4a58929b16aa7e9a2c69f8ee336b0e6e"}
Sep 17 14:03:45.606317 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:45.606192    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e661c74b87f6cfeb5deadf11950c63d4a58929b16aa7e9a2c69f8ee336b0e6e"
Sep 17 14:03:45.606317 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:45.606192    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407pdlg"
Sep 17 14:03:47.997633 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:47.997601    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:03:49.006953 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:49.006917    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:03:50.022505 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:50.022470    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2xxj4"]
Sep 17 14:03:50.022955 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:50.022936    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27f919c4-ad28-4dd2-9a7d-0baae581e785" containerName="extract"
Sep 17 14:03:50.023028 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:50.022960    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="27f919c4-ad28-4dd2-9a7d-0baae581e785" containerName="extract"
Sep 17 14:03:50.023028 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:50.022973    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27f919c4-ad28-4dd2-9a7d-0baae581e785" containerName="util"
Sep 17 14:03:50.023028 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:50.022981    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="27f919c4-ad28-4dd2-9a7d-0baae581e785" containerName="util"
Sep 17 14:03:50.023028 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:50.022994    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fe051b3-4179-4b6d-9360-da9257afae32" containerName="console"
Sep 17 14:03:50.023028 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:50.023003    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fe051b3-4179-4b6d-9360-da9257afae32" containerName="console"
Sep 17 14:03:50.023265 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:50.023030    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27f919c4-ad28-4dd2-9a7d-0baae581e785" containerName="pull"
Sep 17 14:03:50.023265 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:50.023038    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="27f919c4-ad28-4dd2-9a7d-0baae581e785" containerName="pull"
Sep 17 14:03:50.023265 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:50.023131    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fe051b3-4179-4b6d-9360-da9257afae32" containerName="console"
Sep 17 14:03:50.023265 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:50.023145    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="27f919c4-ad28-4dd2-9a7d-0baae581e785" containerName="extract"
Sep 17 14:03:50.026279 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:50.026259    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2xxj4"
Sep 17 14:03:50.029091 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:50.029073    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 17 14:03:50.029386 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:50.029371    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 17 14:03:50.030093 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:50.030080    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-z2wr5\""
Sep 17 14:03:50.040820 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:50.040802    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2xxj4"]
Sep 17 14:03:50.090807 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:50.090784    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0dca25c2-589f-4cfd-a190-4bc73466445d-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-2xxj4\" (UID: \"0dca25c2-589f-4cfd-a190-4bc73466445d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2xxj4"
Sep 17 14:03:50.090911 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:50.090839    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8njw4\" (UniqueName: \"kubernetes.io/projected/0dca25c2-589f-4cfd-a190-4bc73466445d-kube-api-access-8njw4\") pod \"cert-manager-operator-controller-manager-55c444bf86-2xxj4\" (UID: \"0dca25c2-589f-4cfd-a190-4bc73466445d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2xxj4"
Sep 17 14:03:50.191659 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:50.191637    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0dca25c2-589f-4cfd-a190-4bc73466445d-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-2xxj4\" (UID: \"0dca25c2-589f-4cfd-a190-4bc73466445d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2xxj4"
Sep 17 14:03:50.191751 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:50.191684    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8njw4\" (UniqueName: \"kubernetes.io/projected/0dca25c2-589f-4cfd-a190-4bc73466445d-kube-api-access-8njw4\") pod \"cert-manager-operator-controller-manager-55c444bf86-2xxj4\" (UID: \"0dca25c2-589f-4cfd-a190-4bc73466445d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2xxj4"
Sep 17 14:03:50.191976 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:50.191957    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0dca25c2-589f-4cfd-a190-4bc73466445d-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-2xxj4\" (UID: \"0dca25c2-589f-4cfd-a190-4bc73466445d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2xxj4"
Sep 17 14:03:50.199327 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:50.199300    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8njw4\" (UniqueName: \"kubernetes.io/projected/0dca25c2-589f-4cfd-a190-4bc73466445d-kube-api-access-8njw4\") pod \"cert-manager-operator-controller-manager-55c444bf86-2xxj4\" (UID: \"0dca25c2-589f-4cfd-a190-4bc73466445d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2xxj4"
Sep 17 14:03:50.334844 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:50.334813    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2xxj4"
Sep 17 14:03:50.612563 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:50.612482    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2xxj4"]
Sep 17 14:03:50.615770 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:03:50.615733    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dca25c2_589f_4cfd_a190_4bc73466445d.slice/crio-88011bf7687a0fe0a1b297e3a102a5155c5c7b5d2240b7e9152dc227d065696c WatchSource:0}: Error finding container 88011bf7687a0fe0a1b297e3a102a5155c5c7b5d2240b7e9152dc227d065696c: Status 404 returned error can't find the container with id 88011bf7687a0fe0a1b297e3a102a5155c5c7b5d2240b7e9152dc227d065696c
Sep 17 14:03:50.625982 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:50.625959    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2xxj4" event={"ID":"0dca25c2-589f-4cfd-a190-4bc73466445d","Type":"ContainerStarted","Data":"88011bf7687a0fe0a1b297e3a102a5155c5c7b5d2240b7e9152dc227d065696c"}
Sep 17 14:03:51.638339 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:51.638207    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:03:51.738439 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:51.738394    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:03:52.637774 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:52.637740    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2xxj4" event={"ID":"0dca25c2-589f-4cfd-a190-4bc73466445d","Type":"ContainerStarted","Data":"20940b7f3b979e9dbe89401b6d88ea7ace1ca6193611a053fac8255e43e3450e"}
Sep 17 14:03:52.659798 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:52.659738    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2xxj4" podStartSLOduration=0.828739228 podStartE2EDuration="2.659717734s" totalImagesPullingTime="1.830978506s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:03:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:03:52.657563962 +0000 UTC m=+501.599707607" watchObservedRunningTime="2026-09-17 14:03:52.659717734 +0000 UTC m=+501.601861384"
Sep 17 14:03:53.136939 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:53.136897    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:03:53.275934 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:53.275895    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:03:53.367070 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:53.367034    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:03:53.714575 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:53.714543    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-d55cp"]
Sep 17 14:03:53.717902 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:53.717884    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-d55cp"
Sep 17 14:03:53.720145 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:53.720126    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 17 14:03:53.720244 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:53.720214    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-plg8q\""
Sep 17 14:03:53.720998 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:53.720981    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 17 14:03:53.730010 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:53.729987    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-d55cp"]
Sep 17 14:03:53.821263 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:53.821235    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5pp6\" (UniqueName: \"kubernetes.io/projected/7759fdb6-9283-41c3-a9ac-e7e7698a7de5-kube-api-access-s5pp6\") pod \"cert-manager-webhook-6486ffd997-d55cp\" (UID: \"7759fdb6-9283-41c3-a9ac-e7e7698a7de5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-d55cp"
Sep 17 14:03:53.821401 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:53.821299    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7759fdb6-9283-41c3-a9ac-e7e7698a7de5-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-d55cp\" (UID: \"7759fdb6-9283-41c3-a9ac-e7e7698a7de5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-d55cp"
Sep 17 14:03:53.921976 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:53.921948    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7759fdb6-9283-41c3-a9ac-e7e7698a7de5-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-d55cp\" (UID: \"7759fdb6-9283-41c3-a9ac-e7e7698a7de5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-d55cp"
Sep 17 14:03:53.922111 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:53.922003    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5pp6\" (UniqueName: \"kubernetes.io/projected/7759fdb6-9283-41c3-a9ac-e7e7698a7de5-kube-api-access-s5pp6\") pod \"cert-manager-webhook-6486ffd997-d55cp\" (UID: \"7759fdb6-9283-41c3-a9ac-e7e7698a7de5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-d55cp"
Sep 17 14:03:53.930942 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:53.930919    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5pp6\" (UniqueName: \"kubernetes.io/projected/7759fdb6-9283-41c3-a9ac-e7e7698a7de5-kube-api-access-s5pp6\") pod \"cert-manager-webhook-6486ffd997-d55cp\" (UID: \"7759fdb6-9283-41c3-a9ac-e7e7698a7de5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-d55cp"
Sep 17 14:03:53.932037 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:53.932016    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7759fdb6-9283-41c3-a9ac-e7e7698a7de5-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-d55cp\" (UID: \"7759fdb6-9283-41c3-a9ac-e7e7698a7de5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-d55cp"
Sep 17 14:03:54.044054 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:54.043977    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-d55cp"
Sep 17 14:03:54.060011 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:54.059979    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f96n6k"]
Sep 17 14:03:54.063625 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:54.063603    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f96n6k"
Sep 17 14:03:54.066648 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:54.066626    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-d62p6\""
Sep 17 14:03:54.066988 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:54.066971    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 14:03:54.067104 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:54.067087    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 14:03:54.077827 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:54.077805    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f96n6k"]
Sep 17 14:03:54.124204 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:54.124174    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/27a4fa71-60e6-47d8-965f-11da48221098-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f96n6k\" (UID: \"27a4fa71-60e6-47d8-965f-11da48221098\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f96n6k"
Sep 17 14:03:54.124371 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:54.124217    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfj82\" (UniqueName: \"kubernetes.io/projected/27a4fa71-60e6-47d8-965f-11da48221098-kube-api-access-qfj82\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f96n6k\" (UID: \"27a4fa71-60e6-47d8-965f-11da48221098\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f96n6k"
Sep 17 14:03:54.124371 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:54.124356    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/27a4fa71-60e6-47d8-965f-11da48221098-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f96n6k\" (UID: \"27a4fa71-60e6-47d8-965f-11da48221098\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f96n6k"
Sep 17 14:03:54.225270 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:54.225235    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/27a4fa71-60e6-47d8-965f-11da48221098-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f96n6k\" (UID: \"27a4fa71-60e6-47d8-965f-11da48221098\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f96n6k"
Sep 17 14:03:54.225452 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:54.225296    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/27a4fa71-60e6-47d8-965f-11da48221098-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f96n6k\" (UID: \"27a4fa71-60e6-47d8-965f-11da48221098\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f96n6k"
Sep 17 14:03:54.225452 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:54.225324    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfj82\" (UniqueName: \"kubernetes.io/projected/27a4fa71-60e6-47d8-965f-11da48221098-kube-api-access-qfj82\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f96n6k\" (UID: \"27a4fa71-60e6-47d8-965f-11da48221098\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f96n6k"
Sep 17 14:03:54.225653 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:54.225634    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/27a4fa71-60e6-47d8-965f-11da48221098-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f96n6k\" (UID: \"27a4fa71-60e6-47d8-965f-11da48221098\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f96n6k"
Sep 17 14:03:54.225687 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:54.225661    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/27a4fa71-60e6-47d8-965f-11da48221098-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f96n6k\" (UID: \"27a4fa71-60e6-47d8-965f-11da48221098\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f96n6k"
Sep 17 14:03:54.244905 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:54.244873    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfj82\" (UniqueName: \"kubernetes.io/projected/27a4fa71-60e6-47d8-965f-11da48221098-kube-api-access-qfj82\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f96n6k\" (UID: \"27a4fa71-60e6-47d8-965f-11da48221098\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f96n6k"
Sep 17 14:03:54.327202 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:54.327179    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-d55cp"]
Sep 17 14:03:54.329595 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:03:54.329569    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7759fdb6_9283_41c3_a9ac_e7e7698a7de5.slice/crio-b0d56fc10003d7b2c67936b00a49939856915d39d64c03433cd7b877fda61ce5 WatchSource:0}: Error finding container b0d56fc10003d7b2c67936b00a49939856915d39d64c03433cd7b877fda61ce5: Status 404 returned error can't find the container with id b0d56fc10003d7b2c67936b00a49939856915d39d64c03433cd7b877fda61ce5
Sep 17 14:03:54.373572 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:54.373546    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f96n6k"
Sep 17 14:03:54.649303 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:54.649208    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-d55cp" event={"ID":"7759fdb6-9283-41c3-a9ac-e7e7698a7de5","Type":"ContainerStarted","Data":"b0d56fc10003d7b2c67936b00a49939856915d39d64c03433cd7b877fda61ce5"}
Sep 17 14:03:54.649779 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:54.649759    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f96n6k"]
Sep 17 14:03:54.652308 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:03:54.652286    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27a4fa71_60e6_47d8_965f_11da48221098.slice/crio-4e3a9862e2f89b22ec5aed471b7fdd5a36253321647b30a68a7463f7b859bfbf WatchSource:0}: Error finding container 4e3a9862e2f89b22ec5aed471b7fdd5a36253321647b30a68a7463f7b859bfbf: Status 404 returned error can't find the container with id 4e3a9862e2f89b22ec5aed471b7fdd5a36253321647b30a68a7463f7b859bfbf
Sep 17 14:03:55.656539 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:55.656499    2560 generic.go:356] "Generic (PLEG): container finished" podID="27a4fa71-60e6-47d8-965f-11da48221098" containerID="49b17e28200a1c8b97e15f34f83bca658c3c3c40bc8a175b1061b4375b59321d" exitCode=0
Sep 17 14:03:55.657027 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:55.656605    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f96n6k" event={"ID":"27a4fa71-60e6-47d8-965f-11da48221098","Type":"ContainerDied","Data":"49b17e28200a1c8b97e15f34f83bca658c3c3c40bc8a175b1061b4375b59321d"}
Sep 17 14:03:55.657027 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:55.656671    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f96n6k" event={"ID":"27a4fa71-60e6-47d8-965f-11da48221098","Type":"ContainerStarted","Data":"4e3a9862e2f89b22ec5aed471b7fdd5a36253321647b30a68a7463f7b859bfbf"}
Sep 17 14:03:55.894916 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:55.894879    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:03:56.251837 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:56.251800    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:03:57.668829 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:57.668794    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-d55cp" event={"ID":"7759fdb6-9283-41c3-a9ac-e7e7698a7de5","Type":"ContainerStarted","Data":"bc544c4d789150a44b613ad3d98eb835b2b5ceb50f586eebddcf434411b8a358"}
Sep 17 14:03:57.669354 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:57.668929    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-d55cp"
Sep 17 14:03:57.686596 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:57.686546    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-d55cp" podStartSLOduration=2.020299804 podStartE2EDuration="4.686528477s" totalImagesPullingTime="2.666228673s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:03:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:03:57.685798282 +0000 UTC m=+506.627941933" watchObservedRunningTime="2026-09-17 14:03:57.686528477 +0000 UTC m=+506.628672123"
Sep 17 14:03:58.407710 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:58.407673    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:03:58.508143 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:58.508113    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:03:58.675426 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:58.675338    2560 generic.go:356] "Generic (PLEG): container finished" podID="27a4fa71-60e6-47d8-965f-11da48221098" containerID="1ebb7c9d828bb16c35c540ddf97d77350fe6c2a6a5797d09582f1b7d1df88840" exitCode=0
Sep 17 14:03:58.675883 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:58.675420    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f96n6k" event={"ID":"27a4fa71-60e6-47d8-965f-11da48221098","Type":"ContainerDied","Data":"1ebb7c9d828bb16c35c540ddf97d77350fe6c2a6a5797d09582f1b7d1df88840"}
Sep 17 14:03:59.682069 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:59.682038    2560 generic.go:356] "Generic (PLEG): container finished" podID="27a4fa71-60e6-47d8-965f-11da48221098" containerID="5018f60644f1427c41fcc797919d4d7435d4fa6eca8911d9243287442a26a7c0" exitCode=0
Sep 17 14:03:59.682447 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:03:59.682129    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f96n6k" event={"ID":"27a4fa71-60e6-47d8-965f-11da48221098","Type":"ContainerDied","Data":"5018f60644f1427c41fcc797919d4d7435d4fa6eca8911d9243287442a26a7c0"}
Sep 17 14:04:00.804691 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:00.804664    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f96n6k"
Sep 17 14:04:00.882345 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:00.882316    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/27a4fa71-60e6-47d8-965f-11da48221098-util\") pod \"27a4fa71-60e6-47d8-965f-11da48221098\" (UID: \"27a4fa71-60e6-47d8-965f-11da48221098\") "
Sep 17 14:04:00.882497 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:00.882402    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/27a4fa71-60e6-47d8-965f-11da48221098-bundle\") pod \"27a4fa71-60e6-47d8-965f-11da48221098\" (UID: \"27a4fa71-60e6-47d8-965f-11da48221098\") "
Sep 17 14:04:00.882497 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:00.882490    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfj82\" (UniqueName: \"kubernetes.io/projected/27a4fa71-60e6-47d8-965f-11da48221098-kube-api-access-qfj82\") pod \"27a4fa71-60e6-47d8-965f-11da48221098\" (UID: \"27a4fa71-60e6-47d8-965f-11da48221098\") "
Sep 17 14:04:00.882785 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:00.882763    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27a4fa71-60e6-47d8-965f-11da48221098-bundle" (OuterVolumeSpecName: "bundle") pod "27a4fa71-60e6-47d8-965f-11da48221098" (UID: "27a4fa71-60e6-47d8-965f-11da48221098"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:04:00.884373 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:00.884349    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27a4fa71-60e6-47d8-965f-11da48221098-kube-api-access-qfj82" (OuterVolumeSpecName: "kube-api-access-qfj82") pod "27a4fa71-60e6-47d8-965f-11da48221098" (UID: "27a4fa71-60e6-47d8-965f-11da48221098"). InnerVolumeSpecName "kube-api-access-qfj82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:04:00.887255 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:00.887225    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27a4fa71-60e6-47d8-965f-11da48221098-util" (OuterVolumeSpecName: "util") pod "27a4fa71-60e6-47d8-965f-11da48221098" (UID: "27a4fa71-60e6-47d8-965f-11da48221098"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:04:00.984053 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:00.983994    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfj82\" (UniqueName: \"kubernetes.io/projected/27a4fa71-60e6-47d8-965f-11da48221098-kube-api-access-qfj82\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:04:00.984053 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:00.984019    2560 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/27a4fa71-60e6-47d8-965f-11da48221098-util\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:04:00.984053 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:00.984033    2560 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/27a4fa71-60e6-47d8-965f-11da48221098-bundle\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:04:01.699384 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:01.699347    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f96n6k" event={"ID":"27a4fa71-60e6-47d8-965f-11da48221098","Type":"ContainerDied","Data":"4e3a9862e2f89b22ec5aed471b7fdd5a36253321647b30a68a7463f7b859bfbf"}
Sep 17 14:04:01.699567 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:01.699387    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f96n6k"
Sep 17 14:04:01.699567 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:01.699420    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e3a9862e2f89b22ec5aed471b7fdd5a36253321647b30a68a7463f7b859bfbf"
Sep 17 14:04:02.847355 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:02.847322    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:04:02.949353 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:02.949316    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:04:03.678082 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:03.678055    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-d55cp"
Sep 17 14:04:06.642263 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:06.642228    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-pmk6r"]
Sep 17 14:04:06.642831 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:06.642755    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27a4fa71-60e6-47d8-965f-11da48221098" containerName="extract"
Sep 17 14:04:06.642831 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:06.642776    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="27a4fa71-60e6-47d8-965f-11da48221098" containerName="extract"
Sep 17 14:04:06.642831 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:06.642794    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27a4fa71-60e6-47d8-965f-11da48221098" containerName="pull"
Sep 17 14:04:06.642831 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:06.642803    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="27a4fa71-60e6-47d8-965f-11da48221098" containerName="pull"
Sep 17 14:04:06.642831 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:06.642821    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27a4fa71-60e6-47d8-965f-11da48221098" containerName="util"
Sep 17 14:04:06.642831 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:06.642831    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="27a4fa71-60e6-47d8-965f-11da48221098" containerName="util"
Sep 17 14:04:06.643124 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:06.642925    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="27a4fa71-60e6-47d8-965f-11da48221098" containerName="extract"
Sep 17 14:04:06.646280 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:06.646260    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-pmk6r"
Sep 17 14:04:06.648553 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:06.648534    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-ms8lp\""
Sep 17 14:04:06.658710 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:06.658688    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-pmk6r"]
Sep 17 14:04:06.731656 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:06.731631    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr29s\" (UniqueName: \"kubernetes.io/projected/9521e1dd-ec0c-4073-bb5e-65803ffd64c6-kube-api-access-pr29s\") pod \"cert-manager-55b4d954d5-pmk6r\" (UID: \"9521e1dd-ec0c-4073-bb5e-65803ffd64c6\") " pod="cert-manager/cert-manager-55b4d954d5-pmk6r"
Sep 17 14:04:06.731656 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:06.731664    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9521e1dd-ec0c-4073-bb5e-65803ffd64c6-bound-sa-token\") pod \"cert-manager-55b4d954d5-pmk6r\" (UID: \"9521e1dd-ec0c-4073-bb5e-65803ffd64c6\") " pod="cert-manager/cert-manager-55b4d954d5-pmk6r"
Sep 17 14:04:06.832526 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:06.832497    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr29s\" (UniqueName: \"kubernetes.io/projected/9521e1dd-ec0c-4073-bb5e-65803ffd64c6-kube-api-access-pr29s\") pod \"cert-manager-55b4d954d5-pmk6r\" (UID: \"9521e1dd-ec0c-4073-bb5e-65803ffd64c6\") " pod="cert-manager/cert-manager-55b4d954d5-pmk6r"
Sep 17 14:04:06.832526 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:06.832530    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9521e1dd-ec0c-4073-bb5e-65803ffd64c6-bound-sa-token\") pod \"cert-manager-55b4d954d5-pmk6r\" (UID: \"9521e1dd-ec0c-4073-bb5e-65803ffd64c6\") " pod="cert-manager/cert-manager-55b4d954d5-pmk6r"
Sep 17 14:04:06.841310 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:06.841288    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9521e1dd-ec0c-4073-bb5e-65803ffd64c6-bound-sa-token\") pod \"cert-manager-55b4d954d5-pmk6r\" (UID: \"9521e1dd-ec0c-4073-bb5e-65803ffd64c6\") " pod="cert-manager/cert-manager-55b4d954d5-pmk6r"
Sep 17 14:04:06.841420 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:06.841351    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr29s\" (UniqueName: \"kubernetes.io/projected/9521e1dd-ec0c-4073-bb5e-65803ffd64c6-kube-api-access-pr29s\") pod \"cert-manager-55b4d954d5-pmk6r\" (UID: \"9521e1dd-ec0c-4073-bb5e-65803ffd64c6\") " pod="cert-manager/cert-manager-55b4d954d5-pmk6r"
Sep 17 14:04:06.955093 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:06.955021    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-pmk6r"
Sep 17 14:04:07.231803 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:07.231640    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-pmk6r"]
Sep 17 14:04:07.234509 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:04:07.234479    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9521e1dd_ec0c_4073_bb5e_65803ffd64c6.slice/crio-596288a1a9f62388e0d5753243061eb264bb4b368c9f1b3dff76bba08b425d5c WatchSource:0}: Error finding container 596288a1a9f62388e0d5753243061eb264bb4b368c9f1b3dff76bba08b425d5c: Status 404 returned error can't find the container with id 596288a1a9f62388e0d5753243061eb264bb4b368c9f1b3dff76bba08b425d5c
Sep 17 14:04:07.728033 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:07.728000    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-pmk6r" event={"ID":"9521e1dd-ec0c-4073-bb5e-65803ffd64c6","Type":"ContainerStarted","Data":"d12d88c8ed639b6fda2092152c0da69ec05ab862281c38e4507770ebff49cafc"}
Sep 17 14:04:07.728397 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:07.728046    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-pmk6r" event={"ID":"9521e1dd-ec0c-4073-bb5e-65803ffd64c6","Type":"ContainerStarted","Data":"596288a1a9f62388e0d5753243061eb264bb4b368c9f1b3dff76bba08b425d5c"}
Sep 17 14:04:07.745802 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:07.745754    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-pmk6r" podStartSLOduration=1.745738617 podStartE2EDuration="1.745738617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:04:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:04:07.744164688 +0000 UTC m=+516.686308336" watchObservedRunningTime="2026-09-17 14:04:07.745738617 +0000 UTC m=+516.687882263"
Sep 17 14:04:10.014411 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:10.014379    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:04:10.117658 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:10.117619    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:04:12.042941 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:12.042908    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z4knd"]
Sep 17 14:04:12.093013 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:12.092990    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z4knd"]
Sep 17 14:04:12.093148 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:12.093069    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z4knd"
Sep 17 14:04:12.095385 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:12.095368    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 17 14:04:12.095918 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:12.095905    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-cbp88\""
Sep 17 14:04:12.096049 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:12.096026    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 17 14:04:12.171433 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:12.171373    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9npd\" (UniqueName: \"kubernetes.io/projected/531e646a-d3cf-4763-8547-6c48e8bdcc7b-kube-api-access-f9npd\") pod \"openshift-lws-operator-fd8ccff4c-z4knd\" (UID: \"531e646a-d3cf-4763-8547-6c48e8bdcc7b\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z4knd"
Sep 17 14:04:12.171557 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:12.171441    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/531e646a-d3cf-4763-8547-6c48e8bdcc7b-tmp\") pod \"openshift-lws-operator-fd8ccff4c-z4knd\" (UID: \"531e646a-d3cf-4763-8547-6c48e8bdcc7b\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z4knd"
Sep 17 14:04:12.272343 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:12.272312    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/531e646a-d3cf-4763-8547-6c48e8bdcc7b-tmp\") pod \"openshift-lws-operator-fd8ccff4c-z4knd\" (UID: \"531e646a-d3cf-4763-8547-6c48e8bdcc7b\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z4knd"
Sep 17 14:04:12.272542 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:12.272349    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9npd\" (UniqueName: \"kubernetes.io/projected/531e646a-d3cf-4763-8547-6c48e8bdcc7b-kube-api-access-f9npd\") pod \"openshift-lws-operator-fd8ccff4c-z4knd\" (UID: \"531e646a-d3cf-4763-8547-6c48e8bdcc7b\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z4knd"
Sep 17 14:04:12.272770 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:12.272749    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/531e646a-d3cf-4763-8547-6c48e8bdcc7b-tmp\") pod \"openshift-lws-operator-fd8ccff4c-z4knd\" (UID: \"531e646a-d3cf-4763-8547-6c48e8bdcc7b\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z4knd"
Sep 17 14:04:12.281616 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:12.281593    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9npd\" (UniqueName: \"kubernetes.io/projected/531e646a-d3cf-4763-8547-6c48e8bdcc7b-kube-api-access-f9npd\") pod \"openshift-lws-operator-fd8ccff4c-z4knd\" (UID: \"531e646a-d3cf-4763-8547-6c48e8bdcc7b\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z4knd"
Sep 17 14:04:12.403160 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:12.403135    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z4knd"
Sep 17 14:04:12.690551 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:12.690527    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z4knd"]
Sep 17 14:04:12.693927 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:04:12.693902    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod531e646a_d3cf_4763_8547_6c48e8bdcc7b.slice/crio-5f55673a0d6e5c61454581409ca70ddabaf3436baf33cbf86299cc490e5e005a WatchSource:0}: Error finding container 5f55673a0d6e5c61454581409ca70ddabaf3436baf33cbf86299cc490e5e005a: Status 404 returned error can't find the container with id 5f55673a0d6e5c61454581409ca70ddabaf3436baf33cbf86299cc490e5e005a
Sep 17 14:04:12.750909 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:12.750884    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z4knd" event={"ID":"531e646a-d3cf-4763-8547-6c48e8bdcc7b","Type":"ContainerStarted","Data":"5f55673a0d6e5c61454581409ca70ddabaf3436baf33cbf86299cc490e5e005a"}
Sep 17 14:04:15.767164 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:15.767128    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z4knd" event={"ID":"531e646a-d3cf-4763-8547-6c48e8bdcc7b","Type":"ContainerStarted","Data":"b04fc7077bc9e1fa1eeeb0f221a0a13996901925902f7ca30224d556f7d5a35c"}
Sep 17 14:04:15.784708 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:15.784657    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z4knd" podStartSLOduration=1.770077723 podStartE2EDuration="3.784642292s" totalImagesPullingTime="2.014564569s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:04:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:04:15.78297756 +0000 UTC m=+524.725121205" watchObservedRunningTime="2026-09-17 14:04:15.784642292 +0000 UTC m=+524.726785938"
Sep 17 14:04:16.469404 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:16.469371    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:04:16.575247 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:16.575216    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:04:18.543898 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:18.543868    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:04:19.834418 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:19.834384    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp7dxg"]
Sep 17 14:04:19.836757 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:19.836738    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp7dxg"
Sep 17 14:04:19.839086 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:19.839070    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 14:04:19.839577 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:19.839558    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 14:04:19.839658 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:19.839621    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-d62p6\""
Sep 17 14:04:19.848016 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:19.847993    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp7dxg"]
Sep 17 14:04:19.932818 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:19.932792    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8b8764be-bf6c-440a-8edd-2a3f3b598260-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp7dxg\" (UID: \"8b8764be-bf6c-440a-8edd-2a3f3b598260\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp7dxg"
Sep 17 14:04:19.932919 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:19.932832    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8b8764be-bf6c-440a-8edd-2a3f3b598260-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp7dxg\" (UID: \"8b8764be-bf6c-440a-8edd-2a3f3b598260\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp7dxg"
Sep 17 14:04:19.932919 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:19.932854    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwcpt\" (UniqueName: \"kubernetes.io/projected/8b8764be-bf6c-440a-8edd-2a3f3b598260-kube-api-access-zwcpt\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp7dxg\" (UID: \"8b8764be-bf6c-440a-8edd-2a3f3b598260\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp7dxg"
Sep 17 14:04:20.034138 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:20.034112    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8b8764be-bf6c-440a-8edd-2a3f3b598260-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp7dxg\" (UID: \"8b8764be-bf6c-440a-8edd-2a3f3b598260\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp7dxg"
Sep 17 14:04:20.034232 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:20.034156    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8b8764be-bf6c-440a-8edd-2a3f3b598260-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp7dxg\" (UID: \"8b8764be-bf6c-440a-8edd-2a3f3b598260\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp7dxg"
Sep 17 14:04:20.034232 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:20.034176    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwcpt\" (UniqueName: \"kubernetes.io/projected/8b8764be-bf6c-440a-8edd-2a3f3b598260-kube-api-access-zwcpt\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp7dxg\" (UID: \"8b8764be-bf6c-440a-8edd-2a3f3b598260\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp7dxg"
Sep 17 14:04:20.034454 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:20.034435    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8b8764be-bf6c-440a-8edd-2a3f3b598260-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp7dxg\" (UID: \"8b8764be-bf6c-440a-8edd-2a3f3b598260\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp7dxg"
Sep 17 14:04:20.034518 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:20.034497    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8b8764be-bf6c-440a-8edd-2a3f3b598260-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp7dxg\" (UID: \"8b8764be-bf6c-440a-8edd-2a3f3b598260\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp7dxg"
Sep 17 14:04:20.042603 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:20.042579    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwcpt\" (UniqueName: \"kubernetes.io/projected/8b8764be-bf6c-440a-8edd-2a3f3b598260-kube-api-access-zwcpt\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp7dxg\" (UID: \"8b8764be-bf6c-440a-8edd-2a3f3b598260\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp7dxg"
Sep 17 14:04:20.145662 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:20.145607    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp7dxg"
Sep 17 14:04:20.426692 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:20.426623    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp7dxg"]
Sep 17 14:04:20.429815 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:04:20.429789    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b8764be_bf6c_440a_8edd_2a3f3b598260.slice/crio-0253f8d1e181f963848e99e9e86e5b49ff1dc78a449889d6b4d08a7e56c2933f WatchSource:0}: Error finding container 0253f8d1e181f963848e99e9e86e5b49ff1dc78a449889d6b4d08a7e56c2933f: Status 404 returned error can't find the container with id 0253f8d1e181f963848e99e9e86e5b49ff1dc78a449889d6b4d08a7e56c2933f
Sep 17 14:04:20.792944 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:20.792855    2560 generic.go:356] "Generic (PLEG): container finished" podID="8b8764be-bf6c-440a-8edd-2a3f3b598260" containerID="cb68369c590e3691f7f0e28aa6a0755722f54293c67b4caf19e73d50d5e61f32" exitCode=0
Sep 17 14:04:20.793087 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:20.792938    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp7dxg" event={"ID":"8b8764be-bf6c-440a-8edd-2a3f3b598260","Type":"ContainerDied","Data":"cb68369c590e3691f7f0e28aa6a0755722f54293c67b4caf19e73d50d5e61f32"}
Sep 17 14:04:20.793087 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:20.792986    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp7dxg" event={"ID":"8b8764be-bf6c-440a-8edd-2a3f3b598260","Type":"ContainerStarted","Data":"0253f8d1e181f963848e99e9e86e5b49ff1dc78a449889d6b4d08a7e56c2933f"}
Sep 17 14:04:21.798916 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:21.798834    2560 generic.go:356] "Generic (PLEG): container finished" podID="8b8764be-bf6c-440a-8edd-2a3f3b598260" containerID="8751f77c440d70f7d6f4f39720f4a4bc649b7fd76d2103ac24b796c489034194" exitCode=0
Sep 17 14:04:21.798916 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:21.798902    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp7dxg" event={"ID":"8b8764be-bf6c-440a-8edd-2a3f3b598260","Type":"ContainerDied","Data":"8751f77c440d70f7d6f4f39720f4a4bc649b7fd76d2103ac24b796c489034194"}
Sep 17 14:04:22.805512 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:22.805482    2560 generic.go:356] "Generic (PLEG): container finished" podID="8b8764be-bf6c-440a-8edd-2a3f3b598260" containerID="eb2d16515167bf611aec71e31cc7726fa8089fadcee2a4930a53567428ddd5e3" exitCode=0
Sep 17 14:04:22.805910 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:22.805560    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp7dxg" event={"ID":"8b8764be-bf6c-440a-8edd-2a3f3b598260","Type":"ContainerDied","Data":"eb2d16515167bf611aec71e31cc7726fa8089fadcee2a4930a53567428ddd5e3"}
Sep 17 14:04:23.930936 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:23.930914    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp7dxg"
Sep 17 14:04:24.064437 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:24.064360    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8b8764be-bf6c-440a-8edd-2a3f3b598260-util\") pod \"8b8764be-bf6c-440a-8edd-2a3f3b598260\" (UID: \"8b8764be-bf6c-440a-8edd-2a3f3b598260\") "
Sep 17 14:04:24.064437 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:24.064391    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8b8764be-bf6c-440a-8edd-2a3f3b598260-bundle\") pod \"8b8764be-bf6c-440a-8edd-2a3f3b598260\" (UID: \"8b8764be-bf6c-440a-8edd-2a3f3b598260\") "
Sep 17 14:04:24.064638 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:24.064455    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zwcpt\" (UniqueName: \"kubernetes.io/projected/8b8764be-bf6c-440a-8edd-2a3f3b598260-kube-api-access-zwcpt\") pod \"8b8764be-bf6c-440a-8edd-2a3f3b598260\" (UID: \"8b8764be-bf6c-440a-8edd-2a3f3b598260\") "
Sep 17 14:04:24.065133 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:24.065109    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b8764be-bf6c-440a-8edd-2a3f3b598260-bundle" (OuterVolumeSpecName: "bundle") pod "8b8764be-bf6c-440a-8edd-2a3f3b598260" (UID: "8b8764be-bf6c-440a-8edd-2a3f3b598260"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:04:24.066349 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:24.066328    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b8764be-bf6c-440a-8edd-2a3f3b598260-kube-api-access-zwcpt" (OuterVolumeSpecName: "kube-api-access-zwcpt") pod "8b8764be-bf6c-440a-8edd-2a3f3b598260" (UID: "8b8764be-bf6c-440a-8edd-2a3f3b598260"). InnerVolumeSpecName "kube-api-access-zwcpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:04:24.069838 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:24.069803    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b8764be-bf6c-440a-8edd-2a3f3b598260-util" (OuterVolumeSpecName: "util") pod "8b8764be-bf6c-440a-8edd-2a3f3b598260" (UID: "8b8764be-bf6c-440a-8edd-2a3f3b598260"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:04:24.165936 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:24.165907    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwcpt\" (UniqueName: \"kubernetes.io/projected/8b8764be-bf6c-440a-8edd-2a3f3b598260-kube-api-access-zwcpt\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:04:24.165936 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:24.165928    2560 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8b8764be-bf6c-440a-8edd-2a3f3b598260-util\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:04:24.165936 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:24.165936    2560 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8b8764be-bf6c-440a-8edd-2a3f3b598260-bundle\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:04:24.817238 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:24.817214    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp7dxg"
Sep 17 14:04:24.817401 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:24.817212    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp7dxg" event={"ID":"8b8764be-bf6c-440a-8edd-2a3f3b598260","Type":"ContainerDied","Data":"0253f8d1e181f963848e99e9e86e5b49ff1dc78a449889d6b4d08a7e56c2933f"}
Sep 17 14:04:24.817401 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:24.817338    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0253f8d1e181f963848e99e9e86e5b49ff1dc78a449889d6b4d08a7e56c2933f"
Sep 17 14:04:29.833982 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:29.833954    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xmcxs"]
Sep 17 14:04:29.834314 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:29.834275    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b8764be-bf6c-440a-8edd-2a3f3b598260" containerName="pull"
Sep 17 14:04:29.834314 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:29.834286    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b8764be-bf6c-440a-8edd-2a3f3b598260" containerName="pull"
Sep 17 14:04:29.834314 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:29.834293    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b8764be-bf6c-440a-8edd-2a3f3b598260" containerName="extract"
Sep 17 14:04:29.834314 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:29.834298    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b8764be-bf6c-440a-8edd-2a3f3b598260" containerName="extract"
Sep 17 14:04:29.834314 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:29.834305    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b8764be-bf6c-440a-8edd-2a3f3b598260" containerName="util"
Sep 17 14:04:29.834314 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:29.834310    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b8764be-bf6c-440a-8edd-2a3f3b598260" containerName="util"
Sep 17 14:04:29.834522 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:29.834367    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b8764be-bf6c-440a-8edd-2a3f3b598260" containerName="extract"
Sep 17 14:04:29.837597 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:29.837579    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xmcxs"
Sep 17 14:04:29.839883 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:29.839865    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-d62p6\""
Sep 17 14:04:29.839990 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:29.839865    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 14:04:29.840390 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:29.840378    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 14:04:29.850316 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:29.850295    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xmcxs"]
Sep 17 14:04:30.016347 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:30.016317    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gqwp\" (UniqueName: \"kubernetes.io/projected/b4262296-0965-427f-b0ac-1bf81641c63e-kube-api-access-4gqwp\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xmcxs\" (UID: \"b4262296-0965-427f-b0ac-1bf81641c63e\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xmcxs"
Sep 17 14:04:30.016547 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:30.016376    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4262296-0965-427f-b0ac-1bf81641c63e-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xmcxs\" (UID: \"b4262296-0965-427f-b0ac-1bf81641c63e\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xmcxs"
Sep 17 14:04:30.016547 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:30.016486    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4262296-0965-427f-b0ac-1bf81641c63e-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xmcxs\" (UID: \"b4262296-0965-427f-b0ac-1bf81641c63e\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xmcxs"
Sep 17 14:04:30.117706 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:30.117640    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4262296-0965-427f-b0ac-1bf81641c63e-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xmcxs\" (UID: \"b4262296-0965-427f-b0ac-1bf81641c63e\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xmcxs"
Sep 17 14:04:30.117706 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:30.117694    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4262296-0965-427f-b0ac-1bf81641c63e-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xmcxs\" (UID: \"b4262296-0965-427f-b0ac-1bf81641c63e\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xmcxs"
Sep 17 14:04:30.117839 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:30.117740    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gqwp\" (UniqueName: \"kubernetes.io/projected/b4262296-0965-427f-b0ac-1bf81641c63e-kube-api-access-4gqwp\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xmcxs\" (UID: \"b4262296-0965-427f-b0ac-1bf81641c63e\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xmcxs"
Sep 17 14:04:30.118098 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:30.118070    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4262296-0965-427f-b0ac-1bf81641c63e-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xmcxs\" (UID: \"b4262296-0965-427f-b0ac-1bf81641c63e\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xmcxs"
Sep 17 14:04:30.118137 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:30.118093    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4262296-0965-427f-b0ac-1bf81641c63e-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xmcxs\" (UID: \"b4262296-0965-427f-b0ac-1bf81641c63e\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xmcxs"
Sep 17 14:04:30.127051 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:30.127025    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gqwp\" (UniqueName: \"kubernetes.io/projected/b4262296-0965-427f-b0ac-1bf81641c63e-kube-api-access-4gqwp\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xmcxs\" (UID: \"b4262296-0965-427f-b0ac-1bf81641c63e\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xmcxs"
Sep 17 14:04:30.146958 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:30.146942    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xmcxs"
Sep 17 14:04:30.422375 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:30.422310    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xmcxs"]
Sep 17 14:04:30.424441 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:04:30.424413    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4262296_0965_427f_b0ac_1bf81641c63e.slice/crio-064c900b76ca425d91bf55de962a288116b9ac21241abdd6f07abbb0d6f7baf4 WatchSource:0}: Error finding container 064c900b76ca425d91bf55de962a288116b9ac21241abdd6f07abbb0d6f7baf4: Status 404 returned error can't find the container with id 064c900b76ca425d91bf55de962a288116b9ac21241abdd6f07abbb0d6f7baf4
Sep 17 14:04:30.847300 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:30.847266    2560 generic.go:356] "Generic (PLEG): container finished" podID="b4262296-0965-427f-b0ac-1bf81641c63e" containerID="f5b9c11bc49041869a6396b74b55da860f09a55d9ff987755598efb77055e4af" exitCode=0
Sep 17 14:04:30.847686 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:30.847325    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xmcxs" event={"ID":"b4262296-0965-427f-b0ac-1bf81641c63e","Type":"ContainerDied","Data":"f5b9c11bc49041869a6396b74b55da860f09a55d9ff987755598efb77055e4af"}
Sep 17 14:04:30.847686 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:30.847371    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xmcxs" event={"ID":"b4262296-0965-427f-b0ac-1bf81641c63e","Type":"ContainerStarted","Data":"064c900b76ca425d91bf55de962a288116b9ac21241abdd6f07abbb0d6f7baf4"}
Sep 17 14:04:30.918418 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:30.918380    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:04:31.020891 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:31.020853    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:04:31.600508 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:31.600479    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:04:31.856798 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:31.856765    2560 generic.go:356] "Generic (PLEG): container finished" podID="b4262296-0965-427f-b0ac-1bf81641c63e" containerID="c7c1c1603e7cd0d40a9b0ce02366bb5088bae1f3ba0408910cae003cd7780014" exitCode=0
Sep 17 14:04:31.857203 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:31.856826    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xmcxs" event={"ID":"b4262296-0965-427f-b0ac-1bf81641c63e","Type":"ContainerDied","Data":"c7c1c1603e7cd0d40a9b0ce02366bb5088bae1f3ba0408910cae003cd7780014"}
Sep 17 14:04:31.899437 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:31.899413    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:04:32.004988 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:32.004950    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:04:32.090807 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:32.090774    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-fcc687555-sjzts"]
Sep 17 14:04:32.094347 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:32.094326    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-fcc687555-sjzts"
Sep 17 14:04:32.101291 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:32.100390    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 17 14:04:32.101291 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:32.100674    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 17 14:04:32.101291 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:32.100848    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 17 14:04:32.101291 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:32.101021    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 17 14:04:32.101291 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:32.101167    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-gp7tv\""
Sep 17 14:04:32.121153 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:32.121097    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-fcc687555-sjzts"]
Sep 17 14:04:32.225231 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:32.225194    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:04:32.235344 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:32.235323    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ca6e8239-2e88-4499-b547-d2bc9be28568-webhook-cert\") pod \"opendatahub-operator-controller-manager-fcc687555-sjzts\" (UID: \"ca6e8239-2e88-4499-b547-d2bc9be28568\") " pod="opendatahub/opendatahub-operator-controller-manager-fcc687555-sjzts"
Sep 17 14:04:32.235450 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:32.235362    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca6e8239-2e88-4499-b547-d2bc9be28568-tmp\") pod \"opendatahub-operator-controller-manager-fcc687555-sjzts\" (UID: \"ca6e8239-2e88-4499-b547-d2bc9be28568\") " pod="opendatahub/opendatahub-operator-controller-manager-fcc687555-sjzts"
Sep 17 14:04:32.235450 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:32.235436    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ca6e8239-2e88-4499-b547-d2bc9be28568-apiservice-cert\") pod \"opendatahub-operator-controller-manager-fcc687555-sjzts\" (UID: \"ca6e8239-2e88-4499-b547-d2bc9be28568\") " pod="opendatahub/opendatahub-operator-controller-manager-fcc687555-sjzts"
Sep 17 14:04:32.235558 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:32.235499    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ca6e8239-2e88-4499-b547-d2bc9be28568-manifests\") pod \"opendatahub-operator-controller-manager-fcc687555-sjzts\" (UID: \"ca6e8239-2e88-4499-b547-d2bc9be28568\") " pod="opendatahub/opendatahub-operator-controller-manager-fcc687555-sjzts"
Sep 17 14:04:32.235558 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:32.235531    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2j5s\" (UniqueName: \"kubernetes.io/projected/ca6e8239-2e88-4499-b547-d2bc9be28568-kube-api-access-f2j5s\") pod \"opendatahub-operator-controller-manager-fcc687555-sjzts\" (UID: \"ca6e8239-2e88-4499-b547-d2bc9be28568\") " pod="opendatahub/opendatahub-operator-controller-manager-fcc687555-sjzts"
Sep 17 14:04:32.336319 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:32.336284    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ca6e8239-2e88-4499-b547-d2bc9be28568-webhook-cert\") pod \"opendatahub-operator-controller-manager-fcc687555-sjzts\" (UID: \"ca6e8239-2e88-4499-b547-d2bc9be28568\") " pod="opendatahub/opendatahub-operator-controller-manager-fcc687555-sjzts"
Sep 17 14:04:32.336457 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:32.336326    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca6e8239-2e88-4499-b547-d2bc9be28568-tmp\") pod \"opendatahub-operator-controller-manager-fcc687555-sjzts\" (UID: \"ca6e8239-2e88-4499-b547-d2bc9be28568\") " pod="opendatahub/opendatahub-operator-controller-manager-fcc687555-sjzts"
Sep 17 14:04:32.336457 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:32.336364    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ca6e8239-2e88-4499-b547-d2bc9be28568-apiservice-cert\") pod \"opendatahub-operator-controller-manager-fcc687555-sjzts\" (UID: \"ca6e8239-2e88-4499-b547-d2bc9be28568\") " pod="opendatahub/opendatahub-operator-controller-manager-fcc687555-sjzts"
Sep 17 14:04:32.336457 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:32.336395    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ca6e8239-2e88-4499-b547-d2bc9be28568-manifests\") pod \"opendatahub-operator-controller-manager-fcc687555-sjzts\" (UID: \"ca6e8239-2e88-4499-b547-d2bc9be28568\") " pod="opendatahub/opendatahub-operator-controller-manager-fcc687555-sjzts"
Sep 17 14:04:32.336457 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:32.336412    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2j5s\" (UniqueName: \"kubernetes.io/projected/ca6e8239-2e88-4499-b547-d2bc9be28568-kube-api-access-f2j5s\") pod \"opendatahub-operator-controller-manager-fcc687555-sjzts\" (UID: \"ca6e8239-2e88-4499-b547-d2bc9be28568\") " pod="opendatahub/opendatahub-operator-controller-manager-fcc687555-sjzts"
Sep 17 14:04:32.336916 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:32.336887    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ca6e8239-2e88-4499-b547-d2bc9be28568-manifests\") pod \"opendatahub-operator-controller-manager-fcc687555-sjzts\" (UID: \"ca6e8239-2e88-4499-b547-d2bc9be28568\") " pod="opendatahub/opendatahub-operator-controller-manager-fcc687555-sjzts"
Sep 17 14:04:32.338732 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:32.338702    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca6e8239-2e88-4499-b547-d2bc9be28568-tmp\") pod \"opendatahub-operator-controller-manager-fcc687555-sjzts\" (UID: \"ca6e8239-2e88-4499-b547-d2bc9be28568\") " pod="opendatahub/opendatahub-operator-controller-manager-fcc687555-sjzts"
Sep 17 14:04:32.338887 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:32.338865    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ca6e8239-2e88-4499-b547-d2bc9be28568-webhook-cert\") pod \"opendatahub-operator-controller-manager-fcc687555-sjzts\" (UID: \"ca6e8239-2e88-4499-b547-d2bc9be28568\") " pod="opendatahub/opendatahub-operator-controller-manager-fcc687555-sjzts"
Sep 17 14:04:32.339038 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:32.339017    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ca6e8239-2e88-4499-b547-d2bc9be28568-apiservice-cert\") pod \"opendatahub-operator-controller-manager-fcc687555-sjzts\" (UID: \"ca6e8239-2e88-4499-b547-d2bc9be28568\") " pod="opendatahub/opendatahub-operator-controller-manager-fcc687555-sjzts"
Sep 17 14:04:32.345199 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:32.345181    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2j5s\" (UniqueName: \"kubernetes.io/projected/ca6e8239-2e88-4499-b547-d2bc9be28568-kube-api-access-f2j5s\") pod \"opendatahub-operator-controller-manager-fcc687555-sjzts\" (UID: \"ca6e8239-2e88-4499-b547-d2bc9be28568\") " pod="opendatahub/opendatahub-operator-controller-manager-fcc687555-sjzts"
Sep 17 14:04:32.407600 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:32.407548    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-fcc687555-sjzts"
Sep 17 14:04:32.422324 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:32.422299    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:04:32.513868 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:32.513838    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:04:32.614206 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:32.614171    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:04:32.698869 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:32.698842    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-fcc687555-sjzts"]
Sep 17 14:04:32.700792 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:04:32.700768    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca6e8239_2e88_4499_b547_d2bc9be28568.slice/crio-f25df1a1a6a922d32c2b3cdefa1d273e657520c1dfd9e2822a3cc3b90000ba86 WatchSource:0}: Error finding container f25df1a1a6a922d32c2b3cdefa1d273e657520c1dfd9e2822a3cc3b90000ba86: Status 404 returned error can't find the container with id f25df1a1a6a922d32c2b3cdefa1d273e657520c1dfd9e2822a3cc3b90000ba86
Sep 17 14:04:32.710136 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:32.710115    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:04:32.869511 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:32.869456    2560 generic.go:356] "Generic (PLEG): container finished" podID="b4262296-0965-427f-b0ac-1bf81641c63e" containerID="fa3442ec3c3a904a851eee9a63388c5e79f2ff35b7c21855394423be476052f6" exitCode=0
Sep 17 14:04:32.869874 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:32.869485    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xmcxs" event={"ID":"b4262296-0965-427f-b0ac-1bf81641c63e","Type":"ContainerDied","Data":"fa3442ec3c3a904a851eee9a63388c5e79f2ff35b7c21855394423be476052f6"}
Sep 17 14:04:32.870821 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:32.870802    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-fcc687555-sjzts" event={"ID":"ca6e8239-2e88-4499-b547-d2bc9be28568","Type":"ContainerStarted","Data":"f25df1a1a6a922d32c2b3cdefa1d273e657520c1dfd9e2822a3cc3b90000ba86"}
Sep 17 14:04:34.033425 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:34.033400    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xmcxs"
Sep 17 14:04:34.151818 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:34.151731    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4262296-0965-427f-b0ac-1bf81641c63e-util\") pod \"b4262296-0965-427f-b0ac-1bf81641c63e\" (UID: \"b4262296-0965-427f-b0ac-1bf81641c63e\") "
Sep 17 14:04:34.151818 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:34.151775    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4262296-0965-427f-b0ac-1bf81641c63e-bundle\") pod \"b4262296-0965-427f-b0ac-1bf81641c63e\" (UID: \"b4262296-0965-427f-b0ac-1bf81641c63e\") "
Sep 17 14:04:34.152083 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:34.151851    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4gqwp\" (UniqueName: \"kubernetes.io/projected/b4262296-0965-427f-b0ac-1bf81641c63e-kube-api-access-4gqwp\") pod \"b4262296-0965-427f-b0ac-1bf81641c63e\" (UID: \"b4262296-0965-427f-b0ac-1bf81641c63e\") "
Sep 17 14:04:34.152873 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:34.152826    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4262296-0965-427f-b0ac-1bf81641c63e-bundle" (OuterVolumeSpecName: "bundle") pod "b4262296-0965-427f-b0ac-1bf81641c63e" (UID: "b4262296-0965-427f-b0ac-1bf81641c63e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:04:34.154402 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:34.154373    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4262296-0965-427f-b0ac-1bf81641c63e-kube-api-access-4gqwp" (OuterVolumeSpecName: "kube-api-access-4gqwp") pod "b4262296-0965-427f-b0ac-1bf81641c63e" (UID: "b4262296-0965-427f-b0ac-1bf81641c63e"). InnerVolumeSpecName "kube-api-access-4gqwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:04:34.160451 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:34.160422    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4262296-0965-427f-b0ac-1bf81641c63e-util" (OuterVolumeSpecName: "util") pod "b4262296-0965-427f-b0ac-1bf81641c63e" (UID: "b4262296-0965-427f-b0ac-1bf81641c63e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:04:34.253242 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:34.253202    2560 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4262296-0965-427f-b0ac-1bf81641c63e-util\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:04:34.253242 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:34.253240    2560 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4262296-0965-427f-b0ac-1bf81641c63e-bundle\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:04:34.253431 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:34.253256    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gqwp\" (UniqueName: \"kubernetes.io/projected/b4262296-0965-427f-b0ac-1bf81641c63e-kube-api-access-4gqwp\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:04:34.885151 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:34.885114    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xmcxs" event={"ID":"b4262296-0965-427f-b0ac-1bf81641c63e","Type":"ContainerDied","Data":"064c900b76ca425d91bf55de962a288116b9ac21241abdd6f07abbb0d6f7baf4"}
Sep 17 14:04:34.885151 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:34.885144    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xmcxs"
Sep 17 14:04:34.885343 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:34.885163    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="064c900b76ca425d91bf55de962a288116b9ac21241abdd6f07abbb0d6f7baf4"
Sep 17 14:04:36.895336 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:36.895304    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-fcc687555-sjzts" event={"ID":"ca6e8239-2e88-4499-b547-d2bc9be28568","Type":"ContainerStarted","Data":"6e6b7e5984412f5066e258af0651a4bcee90234564a21430c4c25114aa5cb82f"}
Sep 17 14:04:37.901420 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:37.901386    2560 generic.go:356] "Generic (PLEG): container finished" podID="ca6e8239-2e88-4499-b547-d2bc9be28568" containerID="6e6b7e5984412f5066e258af0651a4bcee90234564a21430c4c25114aa5cb82f" exitCode=0
Sep 17 14:04:37.901817 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:37.901483    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-fcc687555-sjzts" event={"ID":"ca6e8239-2e88-4499-b547-d2bc9be28568","Type":"ContainerDied","Data":"6e6b7e5984412f5066e258af0651a4bcee90234564a21430c4c25114aa5cb82f"}
Sep 17 14:04:38.908214 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:38.908177    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-fcc687555-sjzts" event={"ID":"ca6e8239-2e88-4499-b547-d2bc9be28568","Type":"ContainerStarted","Data":"fc22fd74c3308b0dc7d9dbe051a0a02dc40d34cfeafd49e3b242f3cce4136f1c"}
Sep 17 14:04:38.908612 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:38.908274    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-fcc687555-sjzts"
Sep 17 14:04:38.946061 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:38.946012    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-fcc687555-sjzts" podStartSLOduration=2.4223555279999998 podStartE2EDuration="6.945997761s" totalImagesPullingTime="3.484877441s" totalInitContainerRuntime="1.038764792s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:04:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:04:38.944356758 +0000 UTC m=+547.886500404" watchObservedRunningTime="2026-09-17 14:04:38.945997761 +0000 UTC m=+547.888141407"
Sep 17 14:04:39.125009 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:39.124957    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:04:39.225720 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:39.225606    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:04:39.324441 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:39.324400    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:04:39.425386 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:39.425354    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:04:40.364075 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:40.364042    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:04:40.625516 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:40.625415    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:04:41.336577 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:41.336541    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:04:41.436539 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:41.436505    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:04:42.614436 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:42.614402    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:04:42.721227 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:42.721197    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:04:49.914998 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:49.914967    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-fcc687555-sjzts"
Sep 17 14:04:51.470681 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:51.470628    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:04:51.585035 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:51.585000    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:04:51.888984 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:51.888949    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-66d75c69df-wz6ps"]
Sep 17 14:04:51.889290 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:51.889277    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4262296-0965-427f-b0ac-1bf81641c63e" containerName="util"
Sep 17 14:04:51.889290 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:51.889290    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4262296-0965-427f-b0ac-1bf81641c63e" containerName="util"
Sep 17 14:04:51.889377 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:51.889301    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4262296-0965-427f-b0ac-1bf81641c63e" containerName="pull"
Sep 17 14:04:51.889377 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:51.889307    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4262296-0965-427f-b0ac-1bf81641c63e" containerName="pull"
Sep 17 14:04:51.889377 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:51.889351    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4262296-0965-427f-b0ac-1bf81641c63e" containerName="extract"
Sep 17 14:04:51.889377 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:51.889358    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4262296-0965-427f-b0ac-1bf81641c63e" containerName="extract"
Sep 17 14:04:51.889544 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:51.889408    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4262296-0965-427f-b0ac-1bf81641c63e" containerName="extract"
Sep 17 14:04:51.901494 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:51.901446    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-66d75c69df-wz6ps"
Sep 17 14:04:51.905277 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:51.905254    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 17 14:04:51.905422 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:51.905300    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 17 14:04:51.905422 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:51.905309    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-hj4kf\""
Sep 17 14:04:51.916292 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:51.916271    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-66d75c69df-wz6ps"]
Sep 17 14:04:51.990810 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:51.990781    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cdbd220-7e88-4dcf-976f-31b723414622-tls-certs\") pod \"kube-auth-proxy-66d75c69df-wz6ps\" (UID: \"7cdbd220-7e88-4dcf-976f-31b723414622\") " pod="openshift-ingress/kube-auth-proxy-66d75c69df-wz6ps"
Sep 17 14:04:51.990939 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:51.990825    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4t9m\" (UniqueName: \"kubernetes.io/projected/7cdbd220-7e88-4dcf-976f-31b723414622-kube-api-access-p4t9m\") pod \"kube-auth-proxy-66d75c69df-wz6ps\" (UID: \"7cdbd220-7e88-4dcf-976f-31b723414622\") " pod="openshift-ingress/kube-auth-proxy-66d75c69df-wz6ps"
Sep 17 14:04:51.990939 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:51.990877    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7cdbd220-7e88-4dcf-976f-31b723414622-tmp\") pod \"kube-auth-proxy-66d75c69df-wz6ps\" (UID: \"7cdbd220-7e88-4dcf-976f-31b723414622\") " pod="openshift-ingress/kube-auth-proxy-66d75c69df-wz6ps"
Sep 17 14:04:52.091248 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:52.091218    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cdbd220-7e88-4dcf-976f-31b723414622-tls-certs\") pod \"kube-auth-proxy-66d75c69df-wz6ps\" (UID: \"7cdbd220-7e88-4dcf-976f-31b723414622\") " pod="openshift-ingress/kube-auth-proxy-66d75c69df-wz6ps"
Sep 17 14:04:52.091407 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:52.091266    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4t9m\" (UniqueName: \"kubernetes.io/projected/7cdbd220-7e88-4dcf-976f-31b723414622-kube-api-access-p4t9m\") pod \"kube-auth-proxy-66d75c69df-wz6ps\" (UID: \"7cdbd220-7e88-4dcf-976f-31b723414622\") " pod="openshift-ingress/kube-auth-proxy-66d75c69df-wz6ps"
Sep 17 14:04:52.091407 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:52.091296    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7cdbd220-7e88-4dcf-976f-31b723414622-tmp\") pod \"kube-auth-proxy-66d75c69df-wz6ps\" (UID: \"7cdbd220-7e88-4dcf-976f-31b723414622\") " pod="openshift-ingress/kube-auth-proxy-66d75c69df-wz6ps"
Sep 17 14:04:52.093476 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:52.093446    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7cdbd220-7e88-4dcf-976f-31b723414622-tmp\") pod \"kube-auth-proxy-66d75c69df-wz6ps\" (UID: \"7cdbd220-7e88-4dcf-976f-31b723414622\") " pod="openshift-ingress/kube-auth-proxy-66d75c69df-wz6ps"
Sep 17 14:04:52.093736 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:52.093715    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cdbd220-7e88-4dcf-976f-31b723414622-tls-certs\") pod \"kube-auth-proxy-66d75c69df-wz6ps\" (UID: \"7cdbd220-7e88-4dcf-976f-31b723414622\") " pod="openshift-ingress/kube-auth-proxy-66d75c69df-wz6ps"
Sep 17 14:04:52.110647 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:52.110617    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4t9m\" (UniqueName: \"kubernetes.io/projected/7cdbd220-7e88-4dcf-976f-31b723414622-kube-api-access-p4t9m\") pod \"kube-auth-proxy-66d75c69df-wz6ps\" (UID: \"7cdbd220-7e88-4dcf-976f-31b723414622\") " pod="openshift-ingress/kube-auth-proxy-66d75c69df-wz6ps"
Sep 17 14:04:52.211178 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:52.211090    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-66d75c69df-wz6ps"
Sep 17 14:04:52.482248 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:52.482227    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-66d75c69df-wz6ps"]
Sep 17 14:04:52.484185 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:04:52.484159    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cdbd220_7e88_4dcf_976f_31b723414622.slice/crio-79705d80905cf929f0c1a3a9417730e1a357e62a297ab2accf7327dc32be993b WatchSource:0}: Error finding container 79705d80905cf929f0c1a3a9417730e1a357e62a297ab2accf7327dc32be993b: Status 404 returned error can't find the container with id 79705d80905cf929f0c1a3a9417730e1a357e62a297ab2accf7327dc32be993b
Sep 17 14:04:52.517422 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:52.517355    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:04:52.609950 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:52.609920    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:04:52.974379 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:52.974344    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-66d75c69df-wz6ps" event={"ID":"7cdbd220-7e88-4dcf-976f-31b723414622","Type":"ContainerStarted","Data":"79705d80905cf929f0c1a3a9417730e1a357e62a297ab2accf7327dc32be993b"}
Sep 17 14:04:53.378809 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:53.378747    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:04:53.448009 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:53.447974    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:04:54.738739 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:54.738696    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:04:55.999445 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:55.999412    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-66d75c69df-wz6ps" event={"ID":"7cdbd220-7e88-4dcf-976f-31b723414622","Type":"ContainerStarted","Data":"3a5f20e984505d5dc82cdae097e58942cee5e3c903a24db21d2298ac4dd3e203"}
Sep 17 14:04:56.016022 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:56.015977    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-66d75c69df-wz6ps" podStartSLOduration=2.311416899 podStartE2EDuration="5.01596181s" totalImagesPullingTime="2.704544911s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:04:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:04:56.015773359 +0000 UTC m=+564.957917006" watchObservedRunningTime="2026-09-17 14:04:56.01596181 +0000 UTC m=+564.958105456"
Sep 17 14:04:58.487586 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:58.487550    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:04:58.590128 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:58.590096    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:04:59.153239 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:59.153210    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cwmmzm"]
Sep 17 14:04:59.156660 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:59.156640    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cwmmzm"
Sep 17 14:04:59.159188 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:59.159168    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 17 14:04:59.159306 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:59.159168    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-bwjcr\""
Sep 17 14:04:59.171707 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:59.171676    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cwmmzm"]
Sep 17 14:04:59.185432 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:59.185405    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:04:59.251115 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:59.251085    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06ee2c26-2d04-4fa7-8e5d-a70ffe53d2b8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cwmmzm\" (UID: \"06ee2c26-2d04-4fa7-8e5d-a70ffe53d2b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cwmmzm"
Sep 17 14:04:59.251247 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:59.251125    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06ee2c26-2d04-4fa7-8e5d-a70ffe53d2b8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cwmmzm\" (UID: \"06ee2c26-2d04-4fa7-8e5d-a70ffe53d2b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cwmmzm"
Sep 17 14:04:59.251247 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:59.251151    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06ee2c26-2d04-4fa7-8e5d-a70ffe53d2b8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cwmmzm\" (UID: \"06ee2c26-2d04-4fa7-8e5d-a70ffe53d2b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cwmmzm"
Sep 17 14:04:59.251247 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:59.251200    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06ee2c26-2d04-4fa7-8e5d-a70ffe53d2b8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cwmmzm\" (UID: \"06ee2c26-2d04-4fa7-8e5d-a70ffe53d2b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cwmmzm"
Sep 17 14:04:59.251247 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:59.251233    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06ee2c26-2d04-4fa7-8e5d-a70ffe53d2b8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cwmmzm\" (UID: \"06ee2c26-2d04-4fa7-8e5d-a70ffe53d2b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cwmmzm"
Sep 17 14:04:59.251382 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:59.251280    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06ee2c26-2d04-4fa7-8e5d-a70ffe53d2b8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cwmmzm\" (UID: \"06ee2c26-2d04-4fa7-8e5d-a70ffe53d2b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cwmmzm"
Sep 17 14:04:59.251382 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:59.251303    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06ee2c26-2d04-4fa7-8e5d-a70ffe53d2b8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cwmmzm\" (UID: \"06ee2c26-2d04-4fa7-8e5d-a70ffe53d2b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cwmmzm"
Sep 17 14:04:59.251382 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:59.251361    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06ee2c26-2d04-4fa7-8e5d-a70ffe53d2b8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cwmmzm\" (UID: \"06ee2c26-2d04-4fa7-8e5d-a70ffe53d2b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cwmmzm"
Sep 17 14:04:59.251498 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:59.251402    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vhxs\" (UniqueName: \"kubernetes.io/projected/06ee2c26-2d04-4fa7-8e5d-a70ffe53d2b8-kube-api-access-5vhxs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cwmmzm\" (UID: \"06ee2c26-2d04-4fa7-8e5d-a70ffe53d2b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cwmmzm"
Sep 17 14:04:59.352330 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:59.352296    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06ee2c26-2d04-4fa7-8e5d-a70ffe53d2b8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cwmmzm\" (UID: \"06ee2c26-2d04-4fa7-8e5d-a70ffe53d2b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cwmmzm"
Sep 17 14:04:59.352509 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:59.352420    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06ee2c26-2d04-4fa7-8e5d-a70ffe53d2b8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cwmmzm\" (UID: \"06ee2c26-2d04-4fa7-8e5d-a70ffe53d2b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cwmmzm"
Sep 17 14:04:59.352509 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:59.352450    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06ee2c26-2d04-4fa7-8e5d-a70ffe53d2b8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cwmmzm\" (UID: \"06ee2c26-2d04-4fa7-8e5d-a70ffe53d2b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cwmmzm"
Sep 17 14:04:59.352509 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:59.352485    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06ee2c26-2d04-4fa7-8e5d-a70ffe53d2b8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cwmmzm\" (UID: \"06ee2c26-2d04-4fa7-8e5d-a70ffe53d2b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cwmmzm"
Sep 17 14:04:59.352703 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:59.352513    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06ee2c26-2d04-4fa7-8e5d-a70ffe53d2b8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cwmmzm\" (UID: \"06ee2c26-2d04-4fa7-8e5d-a70ffe53d2b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cwmmzm"
Sep 17 14:04:59.352703 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:59.352548    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06ee2c26-2d04-4fa7-8e5d-a70ffe53d2b8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cwmmzm\" (UID: \"06ee2c26-2d04-4fa7-8e5d-a70ffe53d2b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cwmmzm"
Sep 17 14:04:59.352703 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:59.352607    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06ee2c26-2d04-4fa7-8e5d-a70ffe53d2b8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cwmmzm\" (UID: \"06ee2c26-2d04-4fa7-8e5d-a70ffe53d2b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cwmmzm"
Sep 17 14:04:59.352703 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:59.352654    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vhxs\" (UniqueName: \"kubernetes.io/projected/06ee2c26-2d04-4fa7-8e5d-a70ffe53d2b8-kube-api-access-5vhxs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cwmmzm\" (UID: \"06ee2c26-2d04-4fa7-8e5d-a70ffe53d2b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cwmmzm"
Sep 17 14:04:59.352703 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:59.352682    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06ee2c26-2d04-4fa7-8e5d-a70ffe53d2b8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cwmmzm\" (UID: \"06ee2c26-2d04-4fa7-8e5d-a70ffe53d2b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cwmmzm"
Sep 17 14:04:59.352965 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:59.352933    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06ee2c26-2d04-4fa7-8e5d-a70ffe53d2b8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cwmmzm\" (UID: \"06ee2c26-2d04-4fa7-8e5d-a70ffe53d2b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cwmmzm"
Sep 17 14:04:59.353024 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:59.353002    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06ee2c26-2d04-4fa7-8e5d-a70ffe53d2b8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cwmmzm\" (UID: \"06ee2c26-2d04-4fa7-8e5d-a70ffe53d2b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cwmmzm"
Sep 17 14:04:59.353177 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:59.353157    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06ee2c26-2d04-4fa7-8e5d-a70ffe53d2b8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cwmmzm\" (UID: \"06ee2c26-2d04-4fa7-8e5d-a70ffe53d2b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cwmmzm"
Sep 17 14:04:59.353258 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:59.353236    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06ee2c26-2d04-4fa7-8e5d-a70ffe53d2b8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cwmmzm\" (UID: \"06ee2c26-2d04-4fa7-8e5d-a70ffe53d2b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cwmmzm"
Sep 17 14:04:59.353393 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:59.353372    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06ee2c26-2d04-4fa7-8e5d-a70ffe53d2b8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cwmmzm\" (UID: \"06ee2c26-2d04-4fa7-8e5d-a70ffe53d2b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cwmmzm"
Sep 17 14:04:59.355139 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:59.355116    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06ee2c26-2d04-4fa7-8e5d-a70ffe53d2b8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cwmmzm\" (UID: \"06ee2c26-2d04-4fa7-8e5d-a70ffe53d2b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cwmmzm"
Sep 17 14:04:59.355281 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:59.355265    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06ee2c26-2d04-4fa7-8e5d-a70ffe53d2b8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cwmmzm\" (UID: \"06ee2c26-2d04-4fa7-8e5d-a70ffe53d2b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cwmmzm"
Sep 17 14:04:59.373404 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:59.373375    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vhxs\" (UniqueName: \"kubernetes.io/projected/06ee2c26-2d04-4fa7-8e5d-a70ffe53d2b8-kube-api-access-5vhxs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cwmmzm\" (UID: \"06ee2c26-2d04-4fa7-8e5d-a70ffe53d2b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cwmmzm"
Sep 17 14:04:59.374941 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:59.374920    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06ee2c26-2d04-4fa7-8e5d-a70ffe53d2b8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cwmmzm\" (UID: \"06ee2c26-2d04-4fa7-8e5d-a70ffe53d2b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cwmmzm"
Sep 17 14:04:59.416042 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:59.415989    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:04:59.466682 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:59.466662    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cwmmzm"
Sep 17 14:04:59.511801 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:59.511721    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:04:59.780708 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:04:59.780687    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cwmmzm"]
Sep 17 14:04:59.781873 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:04:59.781850    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06ee2c26_2d04_4fa7_8e5d_a70ffe53d2b8.slice/crio-019d03cf4b68c95b04c35942735f3b1e27f2279c4de8c06d62f28053564cb94b WatchSource:0}: Error finding container 019d03cf4b68c95b04c35942735f3b1e27f2279c4de8c06d62f28053564cb94b: Status 404 returned error can't find the container with id 019d03cf4b68c95b04c35942735f3b1e27f2279c4de8c06d62f28053564cb94b
Sep 17 14:05:00.020164 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:00.020127    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cwmmzm" event={"ID":"06ee2c26-2d04-4fa7-8e5d-a70ffe53d2b8","Type":"ContainerStarted","Data":"019d03cf4b68c95b04c35942735f3b1e27f2279c4de8c06d62f28053564cb94b"}
Sep 17 14:05:02.420941 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:02.420904    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-n5wzq"]
Sep 17 14:05:02.428339 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:02.428317    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-n5wzq"
Sep 17 14:05:02.430776 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:02.430750    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-7bsd7\""
Sep 17 14:05:02.430977 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:02.430941    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 17 14:05:02.441219 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:02.441122    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-n5wzq"]
Sep 17 14:05:02.444488 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:02.444386    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:05:02.478586 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:02.478558    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/396ab0f1-ec8d-4a85-891f-7730c439051a-cert\") pod \"odh-model-controller-7998c67dc4-n5wzq\" (UID: \"396ab0f1-ec8d-4a85-891f-7730c439051a\") " pod="opendatahub/odh-model-controller-7998c67dc4-n5wzq"
Sep 17 14:05:02.478734 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:02.478624    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc4nx\" (UniqueName: \"kubernetes.io/projected/396ab0f1-ec8d-4a85-891f-7730c439051a-kube-api-access-rc4nx\") pod \"odh-model-controller-7998c67dc4-n5wzq\" (UID: \"396ab0f1-ec8d-4a85-891f-7730c439051a\") " pod="opendatahub/odh-model-controller-7998c67dc4-n5wzq"
Sep 17 14:05:02.548814 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:02.548776    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:05:02.579810 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:02.579775    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc4nx\" (UniqueName: \"kubernetes.io/projected/396ab0f1-ec8d-4a85-891f-7730c439051a-kube-api-access-rc4nx\") pod \"odh-model-controller-7998c67dc4-n5wzq\" (UID: \"396ab0f1-ec8d-4a85-891f-7730c439051a\") " pod="opendatahub/odh-model-controller-7998c67dc4-n5wzq"
Sep 17 14:05:02.579955 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:02.579912    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/396ab0f1-ec8d-4a85-891f-7730c439051a-cert\") pod \"odh-model-controller-7998c67dc4-n5wzq\" (UID: \"396ab0f1-ec8d-4a85-891f-7730c439051a\") " pod="opendatahub/odh-model-controller-7998c67dc4-n5wzq"
Sep 17 14:05:02.580050 ip-10-0-129-158 kubenswrapper[2560]: E0917 14:05:02.580036    2560 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 17 14:05:02.580125 ip-10-0-129-158 kubenswrapper[2560]: E0917 14:05:02.580112    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/396ab0f1-ec8d-4a85-891f-7730c439051a-cert podName:396ab0f1-ec8d-4a85-891f-7730c439051a nodeName:}" failed. No retries permitted until 2026-09-17 14:05:03.080090028 +0000 UTC m=+572.022233653 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/396ab0f1-ec8d-4a85-891f-7730c439051a-cert") pod "odh-model-controller-7998c67dc4-n5wzq" (UID: "396ab0f1-ec8d-4a85-891f-7730c439051a") : secret "odh-model-controller-webhook-cert" not found
Sep 17 14:05:02.590869 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:02.590849    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc4nx\" (UniqueName: \"kubernetes.io/projected/396ab0f1-ec8d-4a85-891f-7730c439051a-kube-api-access-rc4nx\") pod \"odh-model-controller-7998c67dc4-n5wzq\" (UID: \"396ab0f1-ec8d-4a85-891f-7730c439051a\") " pod="opendatahub/odh-model-controller-7998c67dc4-n5wzq"
Sep 17 14:05:02.643631 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:02.643606    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:05:02.977855 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:02.977802    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:05:03.018331 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:03.018295    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-8ktvg"]
Sep 17 14:05:03.025393 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:03.025373    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-8ktvg"
Sep 17 14:05:03.027585 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:03.027562    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 17 14:05:03.027726 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:03.027565    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-ls8dz\""
Sep 17 14:05:03.032160 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:03.032134    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-8ktvg"]
Sep 17 14:05:03.084656 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:03.084627    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbmjn\" (UniqueName: \"kubernetes.io/projected/f7aa5922-4de4-489a-ac3f-bc61b40034bb-kube-api-access-xbmjn\") pod \"model-serving-api-54d89bd5fc-8ktvg\" (UID: \"f7aa5922-4de4-489a-ac3f-bc61b40034bb\") " pod="opendatahub/model-serving-api-54d89bd5fc-8ktvg"
Sep 17 14:05:03.084805 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:03.084682    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7aa5922-4de4-489a-ac3f-bc61b40034bb-tls-certs\") pod \"model-serving-api-54d89bd5fc-8ktvg\" (UID: \"f7aa5922-4de4-489a-ac3f-bc61b40034bb\") " pod="opendatahub/model-serving-api-54d89bd5fc-8ktvg"
Sep 17 14:05:03.084805 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:03.084757    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/396ab0f1-ec8d-4a85-891f-7730c439051a-cert\") pod \"odh-model-controller-7998c67dc4-n5wzq\" (UID: \"396ab0f1-ec8d-4a85-891f-7730c439051a\") " pod="opendatahub/odh-model-controller-7998c67dc4-n5wzq"
Sep 17 14:05:03.087602 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:03.087577    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/396ab0f1-ec8d-4a85-891f-7730c439051a-cert\") pod \"odh-model-controller-7998c67dc4-n5wzq\" (UID: \"396ab0f1-ec8d-4a85-891f-7730c439051a\") " pod="opendatahub/odh-model-controller-7998c67dc4-n5wzq"
Sep 17 14:05:03.095165 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:03.095131    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:05:03.179288 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:03.179257    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:05:03.179379 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:03.179332    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:05:03.179379 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:03.179360    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:05:03.185368 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:03.185348    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbmjn\" (UniqueName: \"kubernetes.io/projected/f7aa5922-4de4-489a-ac3f-bc61b40034bb-kube-api-access-xbmjn\") pod \"model-serving-api-54d89bd5fc-8ktvg\" (UID: \"f7aa5922-4de4-489a-ac3f-bc61b40034bb\") " pod="opendatahub/model-serving-api-54d89bd5fc-8ktvg"
Sep 17 14:05:03.185415 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:03.185397    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7aa5922-4de4-489a-ac3f-bc61b40034bb-tls-certs\") pod \"model-serving-api-54d89bd5fc-8ktvg\" (UID: \"f7aa5922-4de4-489a-ac3f-bc61b40034bb\") " pod="opendatahub/model-serving-api-54d89bd5fc-8ktvg"
Sep 17 14:05:03.185584 ip-10-0-129-158 kubenswrapper[2560]: E0917 14:05:03.185566    2560 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 17 14:05:03.185643 ip-10-0-129-158 kubenswrapper[2560]: E0917 14:05:03.185632    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7aa5922-4de4-489a-ac3f-bc61b40034bb-tls-certs podName:f7aa5922-4de4-489a-ac3f-bc61b40034bb nodeName:}" failed. No retries permitted until 2026-09-17 14:05:03.68561457 +0000 UTC m=+572.627758208 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/f7aa5922-4de4-489a-ac3f-bc61b40034bb-tls-certs") pod "model-serving-api-54d89bd5fc-8ktvg" (UID: "f7aa5922-4de4-489a-ac3f-bc61b40034bb") : secret "model-serving-api-tls" not found
Sep 17 14:05:03.195205 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:03.195183    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbmjn\" (UniqueName: \"kubernetes.io/projected/f7aa5922-4de4-489a-ac3f-bc61b40034bb-kube-api-access-xbmjn\") pod \"model-serving-api-54d89bd5fc-8ktvg\" (UID: \"f7aa5922-4de4-489a-ac3f-bc61b40034bb\") " pod="opendatahub/model-serving-api-54d89bd5fc-8ktvg"
Sep 17 14:05:03.346533 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:03.346500    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-n5wzq"
Sep 17 14:05:03.622415 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:03.622388    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-n5wzq"]
Sep 17 14:05:03.624774 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:05:03.624746    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod396ab0f1_ec8d_4a85_891f_7730c439051a.slice/crio-3454c88901ed782892aefe2f47ac178274f23245244d68b659094309a4745eae WatchSource:0}: Error finding container 3454c88901ed782892aefe2f47ac178274f23245244d68b659094309a4745eae: Status 404 returned error can't find the container with id 3454c88901ed782892aefe2f47ac178274f23245244d68b659094309a4745eae
Sep 17 14:05:03.688872 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:03.688844    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7aa5922-4de4-489a-ac3f-bc61b40034bb-tls-certs\") pod \"model-serving-api-54d89bd5fc-8ktvg\" (UID: \"f7aa5922-4de4-489a-ac3f-bc61b40034bb\") " pod="opendatahub/model-serving-api-54d89bd5fc-8ktvg"
Sep 17 14:05:03.691186 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:03.691166    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7aa5922-4de4-489a-ac3f-bc61b40034bb-tls-certs\") pod \"model-serving-api-54d89bd5fc-8ktvg\" (UID: \"f7aa5922-4de4-489a-ac3f-bc61b40034bb\") " pod="opendatahub/model-serving-api-54d89bd5fc-8ktvg"
Sep 17 14:05:03.939928 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:03.939849    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-8ktvg"
Sep 17 14:05:04.044069 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:04.044035    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-n5wzq" event={"ID":"396ab0f1-ec8d-4a85-891f-7730c439051a","Type":"ContainerStarted","Data":"3454c88901ed782892aefe2f47ac178274f23245244d68b659094309a4745eae"}
Sep 17 14:05:04.045668 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:04.045636    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cwmmzm" event={"ID":"06ee2c26-2d04-4fa7-8e5d-a70ffe53d2b8","Type":"ContainerStarted","Data":"57483b40d8ea232589c5ca9a18b86c1df3b843a222a54f1d3c1f9ff4d87c59c6"}
Sep 17 14:05:04.077223 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:04.077197    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:05:04.214935 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:04.214841    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cwmmzm" podStartSLOduration=1.8197694850000001 podStartE2EDuration="5.214823977s" totalImagesPullingTime="3.395054492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:04:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:05:04.069064355 +0000 UTC m=+573.011208002" watchObservedRunningTime="2026-09-17 14:05:04.214823977 +0000 UTC m=+573.156967622"
Sep 17 14:05:04.215920 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:04.215899    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-8ktvg"]
Sep 17 14:05:04.217790 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:05:04.217755    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7aa5922_4de4_489a_ac3f_bc61b40034bb.slice/crio-18d37a8ddfeccadd99061bc0763d8e8111162c78e94ef33cd9a41b7e10c0bfc2 WatchSource:0}: Error finding container 18d37a8ddfeccadd99061bc0763d8e8111162c78e94ef33cd9a41b7e10c0bfc2: Status 404 returned error can't find the container with id 18d37a8ddfeccadd99061bc0763d8e8111162c78e94ef33cd9a41b7e10c0bfc2
Sep 17 14:05:04.467845 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:04.467775    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cwmmzm"
Sep 17 14:05:04.472025 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:04.472003    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cwmmzm"
Sep 17 14:05:05.056409 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:05.056282    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-8ktvg" event={"ID":"f7aa5922-4de4-489a-ac3f-bc61b40034bb","Type":"ContainerStarted","Data":"18d37a8ddfeccadd99061bc0763d8e8111162c78e94ef33cd9a41b7e10c0bfc2"}
Sep 17 14:05:05.056409 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:05.056349    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cwmmzm"
Sep 17 14:05:05.058017 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:05.057994    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cwmmzm"
Sep 17 14:05:08.073833 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:08.073800    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-n5wzq" event={"ID":"396ab0f1-ec8d-4a85-891f-7730c439051a","Type":"ContainerStarted","Data":"945d46a70c876c87fc30e721323129fb121c9bdf672f7dafda0dcec2097ee487"}
Sep 17 14:05:08.074253 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:08.073927    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-n5wzq"
Sep 17 14:05:08.075446 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:08.075424    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-8ktvg" event={"ID":"f7aa5922-4de4-489a-ac3f-bc61b40034bb","Type":"ContainerStarted","Data":"361a81b74a69d98e053f319ab63b6f92bc00c12f2893b309dc65a254ffab35ac"}
Sep 17 14:05:08.075579 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:08.075564    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-8ktvg"
Sep 17 14:05:08.095175 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:08.095139    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-n5wzq" podStartSLOduration=2.262205428 podStartE2EDuration="6.095126197s" totalImagesPullingTime="3.832920769s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:05:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:05:08.093028308 +0000 UTC m=+577.035171955" watchObservedRunningTime="2026-09-17 14:05:08.095126197 +0000 UTC m=+577.037269887"
Sep 17 14:05:08.117348 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:08.117308    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-8ktvg" podStartSLOduration=1.880136549 podStartE2EDuration="5.117292231s" totalImagesPullingTime="3.237155682s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:05:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:05:08.114709331 +0000 UTC m=+577.056852978" watchObservedRunningTime="2026-09-17 14:05:08.117292231 +0000 UTC m=+577.059436123"
Sep 17 14:05:10.765002 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:10.764968    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-69b649694d-jn5nd"]
Sep 17 14:05:10.769110 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:10.769086    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-69b649694d-jn5nd"
Sep 17 14:05:10.773315 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:10.773289    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 17 14:05:10.773957 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:10.773937    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 17 14:05:10.774055 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:10.773957    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-vqbp7\""
Sep 17 14:05:10.777494 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:10.777475    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 17 14:05:10.794241 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:10.794220    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-69b649694d-jn5nd"]
Sep 17 14:05:10.860114 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:10.860086    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c26ba51c-6096-49ae-9933-3c1f4398d5cf-metrics-cert\") pod \"lws-controller-manager-69b649694d-jn5nd\" (UID: \"c26ba51c-6096-49ae-9933-3c1f4398d5cf\") " pod="openshift-lws-operator/lws-controller-manager-69b649694d-jn5nd"
Sep 17 14:05:10.860114 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:10.860115    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c26ba51c-6096-49ae-9933-3c1f4398d5cf-manager-config\") pod \"lws-controller-manager-69b649694d-jn5nd\" (UID: \"c26ba51c-6096-49ae-9933-3c1f4398d5cf\") " pod="openshift-lws-operator/lws-controller-manager-69b649694d-jn5nd"
Sep 17 14:05:10.860295 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:10.860150    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q77k8\" (UniqueName: \"kubernetes.io/projected/c26ba51c-6096-49ae-9933-3c1f4398d5cf-kube-api-access-q77k8\") pod \"lws-controller-manager-69b649694d-jn5nd\" (UID: \"c26ba51c-6096-49ae-9933-3c1f4398d5cf\") " pod="openshift-lws-operator/lws-controller-manager-69b649694d-jn5nd"
Sep 17 14:05:10.860295 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:10.860167    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c26ba51c-6096-49ae-9933-3c1f4398d5cf-cert\") pod \"lws-controller-manager-69b649694d-jn5nd\" (UID: \"c26ba51c-6096-49ae-9933-3c1f4398d5cf\") " pod="openshift-lws-operator/lws-controller-manager-69b649694d-jn5nd"
Sep 17 14:05:10.960828 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:10.960800    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q77k8\" (UniqueName: \"kubernetes.io/projected/c26ba51c-6096-49ae-9933-3c1f4398d5cf-kube-api-access-q77k8\") pod \"lws-controller-manager-69b649694d-jn5nd\" (UID: \"c26ba51c-6096-49ae-9933-3c1f4398d5cf\") " pod="openshift-lws-operator/lws-controller-manager-69b649694d-jn5nd"
Sep 17 14:05:10.960944 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:10.960830    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c26ba51c-6096-49ae-9933-3c1f4398d5cf-cert\") pod \"lws-controller-manager-69b649694d-jn5nd\" (UID: \"c26ba51c-6096-49ae-9933-3c1f4398d5cf\") " pod="openshift-lws-operator/lws-controller-manager-69b649694d-jn5nd"
Sep 17 14:05:10.960944 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:10.960902    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c26ba51c-6096-49ae-9933-3c1f4398d5cf-metrics-cert\") pod \"lws-controller-manager-69b649694d-jn5nd\" (UID: \"c26ba51c-6096-49ae-9933-3c1f4398d5cf\") " pod="openshift-lws-operator/lws-controller-manager-69b649694d-jn5nd"
Sep 17 14:05:10.960944 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:10.960919    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c26ba51c-6096-49ae-9933-3c1f4398d5cf-manager-config\") pod \"lws-controller-manager-69b649694d-jn5nd\" (UID: \"c26ba51c-6096-49ae-9933-3c1f4398d5cf\") " pod="openshift-lws-operator/lws-controller-manager-69b649694d-jn5nd"
Sep 17 14:05:10.961625 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:10.961604    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c26ba51c-6096-49ae-9933-3c1f4398d5cf-manager-config\") pod \"lws-controller-manager-69b649694d-jn5nd\" (UID: \"c26ba51c-6096-49ae-9933-3c1f4398d5cf\") " pod="openshift-lws-operator/lws-controller-manager-69b649694d-jn5nd"
Sep 17 14:05:10.963253 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:10.963225    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c26ba51c-6096-49ae-9933-3c1f4398d5cf-metrics-cert\") pod \"lws-controller-manager-69b649694d-jn5nd\" (UID: \"c26ba51c-6096-49ae-9933-3c1f4398d5cf\") " pod="openshift-lws-operator/lws-controller-manager-69b649694d-jn5nd"
Sep 17 14:05:10.963347 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:10.963297    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c26ba51c-6096-49ae-9933-3c1f4398d5cf-cert\") pod \"lws-controller-manager-69b649694d-jn5nd\" (UID: \"c26ba51c-6096-49ae-9933-3c1f4398d5cf\") " pod="openshift-lws-operator/lws-controller-manager-69b649694d-jn5nd"
Sep 17 14:05:10.971793 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:10.971774    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q77k8\" (UniqueName: \"kubernetes.io/projected/c26ba51c-6096-49ae-9933-3c1f4398d5cf-kube-api-access-q77k8\") pod \"lws-controller-manager-69b649694d-jn5nd\" (UID: \"c26ba51c-6096-49ae-9933-3c1f4398d5cf\") " pod="openshift-lws-operator/lws-controller-manager-69b649694d-jn5nd"
Sep 17 14:05:11.080902 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:11.080862    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-69b649694d-jn5nd"
Sep 17 14:05:11.368341 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:11.368268    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-69b649694d-jn5nd"]
Sep 17 14:05:11.371181 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:05:11.371152    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc26ba51c_6096_49ae_9933_3c1f4398d5cf.slice/crio-b81a43d3a6a6040f4e493d052118dd40e01551f3f7cab5b55cbb8f4e8e195f32 WatchSource:0}: Error finding container b81a43d3a6a6040f4e493d052118dd40e01551f3f7cab5b55cbb8f4e8e195f32: Status 404 returned error can't find the container with id b81a43d3a6a6040f4e493d052118dd40e01551f3f7cab5b55cbb8f4e8e195f32
Sep 17 14:05:12.094775 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:12.094734    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-69b649694d-jn5nd" event={"ID":"c26ba51c-6096-49ae-9933-3c1f4398d5cf","Type":"ContainerStarted","Data":"b81a43d3a6a6040f4e493d052118dd40e01551f3f7cab5b55cbb8f4e8e195f32"}
Sep 17 14:05:12.336142 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:12.336102    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:05:12.489735 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:12.489686    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:05:12.599552 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:12.599453    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:05:13.045601 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:13.045512    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:05:13.691521 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:13.691438    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:05:13.820833 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:13.820798    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:05:13.949542 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:13.949396    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:05:14.108168 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:14.108134    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-69b649694d-jn5nd" event={"ID":"c26ba51c-6096-49ae-9933-3c1f4398d5cf","Type":"ContainerStarted","Data":"33bc200aa5275d5392bd018b4a6b537566dd6ae2ed97a19df929d9f72b0c4af1"}
Sep 17 14:05:14.108346 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:14.108277    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-69b649694d-jn5nd"
Sep 17 14:05:14.132711 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:14.132668    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-69b649694d-jn5nd" podStartSLOduration=2.196186769 podStartE2EDuration="4.132653235s" totalImagesPullingTime="1.936466466s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:05:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:05:14.129860853 +0000 UTC m=+583.072004499" watchObservedRunningTime="2026-09-17 14:05:14.132653235 +0000 UTC m=+583.074796880"
Sep 17 14:05:19.082809 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:19.082777    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-n5wzq"
Sep 17 14:05:19.085728 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:19.085703    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-8ktvg"
Sep 17 14:05:25.116506 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:05:25.116477    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-69b649694d-jn5nd"
Sep 17 14:06:03.605957 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:03.605854    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:06:03.700125 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:03.700097    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:06:08.126367 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:08.126335    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfhbgc"]
Sep 17 14:06:08.130063 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:08.130046    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfhbgc"
Sep 17 14:06:08.132525 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:08.132504    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 14:06:08.132622 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:08.132572    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 14:06:08.133079 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:08.133063    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-d62p6\""
Sep 17 14:06:08.149208 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:08.149178    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfhbgc"]
Sep 17 14:06:08.223328 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:08.223296    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zlfbp"]
Sep 17 14:06:08.225165 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:08.225141    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psxl6\" (UniqueName: \"kubernetes.io/projected/6e5ee2d3-a52a-499b-a1d8-b707f3c6676a-kube-api-access-psxl6\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfhbgc\" (UID: \"6e5ee2d3-a52a-499b-a1d8-b707f3c6676a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfhbgc"
Sep 17 14:06:08.225235 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:08.225197    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6e5ee2d3-a52a-499b-a1d8-b707f3c6676a-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfhbgc\" (UID: \"6e5ee2d3-a52a-499b-a1d8-b707f3c6676a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfhbgc"
Sep 17 14:06:08.225299 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:08.225279    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6e5ee2d3-a52a-499b-a1d8-b707f3c6676a-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfhbgc\" (UID: \"6e5ee2d3-a52a-499b-a1d8-b707f3c6676a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfhbgc"
Sep 17 14:06:08.227025 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:08.227010    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zlfbp"
Sep 17 14:06:08.236150 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:08.236129    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zlfbp"]
Sep 17 14:06:08.326010 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:08.325978    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6e5ee2d3-a52a-499b-a1d8-b707f3c6676a-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfhbgc\" (UID: \"6e5ee2d3-a52a-499b-a1d8-b707f3c6676a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfhbgc"
Sep 17 14:06:08.326183 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:08.326019    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/100093d5-c861-4ba1-a956-6fb0cc5dcadd-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zlfbp\" (UID: \"100093d5-c861-4ba1-a956-6fb0cc5dcadd\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zlfbp"
Sep 17 14:06:08.326183 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:08.326060    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/100093d5-c861-4ba1-a956-6fb0cc5dcadd-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zlfbp\" (UID: \"100093d5-c861-4ba1-a956-6fb0cc5dcadd\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zlfbp"
Sep 17 14:06:08.326183 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:08.326089    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdthh\" (UniqueName: \"kubernetes.io/projected/100093d5-c861-4ba1-a956-6fb0cc5dcadd-kube-api-access-wdthh\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zlfbp\" (UID: \"100093d5-c861-4ba1-a956-6fb0cc5dcadd\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zlfbp"
Sep 17 14:06:08.326183 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:08.326119    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psxl6\" (UniqueName: \"kubernetes.io/projected/6e5ee2d3-a52a-499b-a1d8-b707f3c6676a-kube-api-access-psxl6\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfhbgc\" (UID: \"6e5ee2d3-a52a-499b-a1d8-b707f3c6676a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfhbgc"
Sep 17 14:06:08.326318 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:08.326198    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6e5ee2d3-a52a-499b-a1d8-b707f3c6676a-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfhbgc\" (UID: \"6e5ee2d3-a52a-499b-a1d8-b707f3c6676a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfhbgc"
Sep 17 14:06:08.326524 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:08.326506    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6e5ee2d3-a52a-499b-a1d8-b707f3c6676a-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfhbgc\" (UID: \"6e5ee2d3-a52a-499b-a1d8-b707f3c6676a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfhbgc"
Sep 17 14:06:08.326573 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:08.326527    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6e5ee2d3-a52a-499b-a1d8-b707f3c6676a-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfhbgc\" (UID: \"6e5ee2d3-a52a-499b-a1d8-b707f3c6676a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfhbgc"
Sep 17 14:06:08.335148 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:08.335123    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psxl6\" (UniqueName: \"kubernetes.io/projected/6e5ee2d3-a52a-499b-a1d8-b707f3c6676a-kube-api-access-psxl6\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfhbgc\" (UID: \"6e5ee2d3-a52a-499b-a1d8-b707f3c6676a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfhbgc"
Sep 17 14:06:08.339433 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:08.339412    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6487kn"]
Sep 17 14:06:08.343260 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:08.343243    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6487kn"
Sep 17 14:06:08.351561 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:08.351540    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6487kn"]
Sep 17 14:06:08.427312 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:08.427239    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/100093d5-c861-4ba1-a956-6fb0cc5dcadd-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zlfbp\" (UID: \"100093d5-c861-4ba1-a956-6fb0cc5dcadd\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zlfbp"
Sep 17 14:06:08.427312 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:08.427282    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eee1c619-1676-4c44-b9d6-c01ae83c344c-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6487kn\" (UID: \"eee1c619-1676-4c44-b9d6-c01ae83c344c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6487kn"
Sep 17 14:06:08.427312 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:08.427301    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/100093d5-c861-4ba1-a956-6fb0cc5dcadd-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zlfbp\" (UID: \"100093d5-c861-4ba1-a956-6fb0cc5dcadd\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zlfbp"
Sep 17 14:06:08.427566 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:08.427326    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r94tw\" (UniqueName: \"kubernetes.io/projected/eee1c619-1676-4c44-b9d6-c01ae83c344c-kube-api-access-r94tw\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6487kn\" (UID: \"eee1c619-1676-4c44-b9d6-c01ae83c344c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6487kn"
Sep 17 14:06:08.427566 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:08.427353    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdthh\" (UniqueName: \"kubernetes.io/projected/100093d5-c861-4ba1-a956-6fb0cc5dcadd-kube-api-access-wdthh\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zlfbp\" (UID: \"100093d5-c861-4ba1-a956-6fb0cc5dcadd\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zlfbp"
Sep 17 14:06:08.427566 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:08.427373    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eee1c619-1676-4c44-b9d6-c01ae83c344c-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6487kn\" (UID: \"eee1c619-1676-4c44-b9d6-c01ae83c344c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6487kn"
Sep 17 14:06:08.427664 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:08.427610    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/100093d5-c861-4ba1-a956-6fb0cc5dcadd-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zlfbp\" (UID: \"100093d5-c861-4ba1-a956-6fb0cc5dcadd\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zlfbp"
Sep 17 14:06:08.427664 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:08.427647    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/100093d5-c861-4ba1-a956-6fb0cc5dcadd-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zlfbp\" (UID: \"100093d5-c861-4ba1-a956-6fb0cc5dcadd\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zlfbp"
Sep 17 14:06:08.435492 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:08.435455    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdthh\" (UniqueName: \"kubernetes.io/projected/100093d5-c861-4ba1-a956-6fb0cc5dcadd-kube-api-access-wdthh\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zlfbp\" (UID: \"100093d5-c861-4ba1-a956-6fb0cc5dcadd\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zlfbp"
Sep 17 14:06:08.448988 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:08.448968    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfhbgc"
Sep 17 14:06:08.461777 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:08.461759    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a65g8q"]
Sep 17 14:06:08.465573 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:08.465555    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a65g8q"
Sep 17 14:06:08.475984 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:08.475956    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a65g8q"]
Sep 17 14:06:08.528387 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:08.528358    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eee1c619-1676-4c44-b9d6-c01ae83c344c-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6487kn\" (UID: \"eee1c619-1676-4c44-b9d6-c01ae83c344c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6487kn"
Sep 17 14:06:08.528547 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:08.528444    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce73b866-39ce-41e2-b34f-831fcee4fb0e-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a65g8q\" (UID: \"ce73b866-39ce-41e2-b34f-831fcee4fb0e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a65g8q"
Sep 17 14:06:08.528547 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:08.528485    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ce73b866-39ce-41e2-b34f-831fcee4fb0e-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a65g8q\" (UID: \"ce73b866-39ce-41e2-b34f-831fcee4fb0e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a65g8q"
Sep 17 14:06:08.528650 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:08.528601    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnckq\" (UniqueName: \"kubernetes.io/projected/ce73b866-39ce-41e2-b34f-831fcee4fb0e-kube-api-access-lnckq\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a65g8q\" (UID: \"ce73b866-39ce-41e2-b34f-831fcee4fb0e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a65g8q"
Sep 17 14:06:08.528650 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:08.528641    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eee1c619-1676-4c44-b9d6-c01ae83c344c-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6487kn\" (UID: \"eee1c619-1676-4c44-b9d6-c01ae83c344c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6487kn"
Sep 17 14:06:08.528739 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:08.528663    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r94tw\" (UniqueName: \"kubernetes.io/projected/eee1c619-1676-4c44-b9d6-c01ae83c344c-kube-api-access-r94tw\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6487kn\" (UID: \"eee1c619-1676-4c44-b9d6-c01ae83c344c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6487kn"
Sep 17 14:06:08.528782 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:08.528765    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eee1c619-1676-4c44-b9d6-c01ae83c344c-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6487kn\" (UID: \"eee1c619-1676-4c44-b9d6-c01ae83c344c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6487kn"
Sep 17 14:06:08.528954 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:08.528936    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eee1c619-1676-4c44-b9d6-c01ae83c344c-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6487kn\" (UID: \"eee1c619-1676-4c44-b9d6-c01ae83c344c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6487kn"
Sep 17 14:06:08.536744 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:08.536712    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zlfbp"
Sep 17 14:06:08.539095 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:08.539069    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r94tw\" (UniqueName: \"kubernetes.io/projected/eee1c619-1676-4c44-b9d6-c01ae83c344c-kube-api-access-r94tw\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6487kn\" (UID: \"eee1c619-1676-4c44-b9d6-c01ae83c344c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6487kn"
Sep 17 14:06:08.629092 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:08.629066    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce73b866-39ce-41e2-b34f-831fcee4fb0e-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a65g8q\" (UID: \"ce73b866-39ce-41e2-b34f-831fcee4fb0e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a65g8q"
Sep 17 14:06:08.629209 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:08.629096    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ce73b866-39ce-41e2-b34f-831fcee4fb0e-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a65g8q\" (UID: \"ce73b866-39ce-41e2-b34f-831fcee4fb0e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a65g8q"
Sep 17 14:06:08.629279 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:08.629236    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnckq\" (UniqueName: \"kubernetes.io/projected/ce73b866-39ce-41e2-b34f-831fcee4fb0e-kube-api-access-lnckq\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a65g8q\" (UID: \"ce73b866-39ce-41e2-b34f-831fcee4fb0e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a65g8q"
Sep 17 14:06:08.629688 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:08.629481    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ce73b866-39ce-41e2-b34f-831fcee4fb0e-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a65g8q\" (UID: \"ce73b866-39ce-41e2-b34f-831fcee4fb0e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a65g8q"
Sep 17 14:06:08.629688 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:08.629537    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce73b866-39ce-41e2-b34f-831fcee4fb0e-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a65g8q\" (UID: \"ce73b866-39ce-41e2-b34f-831fcee4fb0e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a65g8q"
Sep 17 14:06:08.637663 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:08.637644    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnckq\" (UniqueName: \"kubernetes.io/projected/ce73b866-39ce-41e2-b34f-831fcee4fb0e-kube-api-access-lnckq\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a65g8q\" (UID: \"ce73b866-39ce-41e2-b34f-831fcee4fb0e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a65g8q"
Sep 17 14:06:08.653117 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:08.653096    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6487kn"
Sep 17 14:06:08.727422 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:08.727397    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfhbgc"]
Sep 17 14:06:08.730245 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:06:08.730210    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e5ee2d3_a52a_499b_a1d8_b707f3c6676a.slice/crio-ea790eacc4813eaa7dd0e6e6808887cbe7b3ee0e87c0e8079bdaec52dec8e015 WatchSource:0}: Error finding container ea790eacc4813eaa7dd0e6e6808887cbe7b3ee0e87c0e8079bdaec52dec8e015: Status 404 returned error can't find the container with id ea790eacc4813eaa7dd0e6e6808887cbe7b3ee0e87c0e8079bdaec52dec8e015
Sep 17 14:06:08.783699 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:08.783681    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a65g8q"
Sep 17 14:06:08.811307 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:08.811162    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zlfbp"]
Sep 17 14:06:08.813612 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:06:08.813572    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod100093d5_c861_4ba1_a956_6fb0cc5dcadd.slice/crio-31f213ccf973dd0b299fd030baa1dc073e916dcbd13540eb80e628ac49214149 WatchSource:0}: Error finding container 31f213ccf973dd0b299fd030baa1dc073e916dcbd13540eb80e628ac49214149: Status 404 returned error can't find the container with id 31f213ccf973dd0b299fd030baa1dc073e916dcbd13540eb80e628ac49214149
Sep 17 14:06:08.928049 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:08.928025    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6487kn"]
Sep 17 14:06:08.929208 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:06:08.929188    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeee1c619_1676_4c44_b9d6_c01ae83c344c.slice/crio-21d5ece5c91db1fdd1b117d023d06b5277ec08fa407c43d185dcac78222c9996 WatchSource:0}: Error finding container 21d5ece5c91db1fdd1b117d023d06b5277ec08fa407c43d185dcac78222c9996: Status 404 returned error can't find the container with id 21d5ece5c91db1fdd1b117d023d06b5277ec08fa407c43d185dcac78222c9996
Sep 17 14:06:09.061005 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:09.060981    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a65g8q"]
Sep 17 14:06:09.075262 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:06:09.075234    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce73b866_39ce_41e2_b34f_831fcee4fb0e.slice/crio-6056ab8413ebb54890db7bc90d3787220860e8aae088ce604450ed20b082f3b1 WatchSource:0}: Error finding container 6056ab8413ebb54890db7bc90d3787220860e8aae088ce604450ed20b082f3b1: Status 404 returned error can't find the container with id 6056ab8413ebb54890db7bc90d3787220860e8aae088ce604450ed20b082f3b1
Sep 17 14:06:09.399184 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:09.399154    2560 generic.go:356] "Generic (PLEG): container finished" podID="ce73b866-39ce-41e2-b34f-831fcee4fb0e" containerID="db33808df07b4ab5ff120bc8cf85057840450381a9874fc4c569f5f9b280ced0" exitCode=0
Sep 17 14:06:09.399672 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:09.399224    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a65g8q" event={"ID":"ce73b866-39ce-41e2-b34f-831fcee4fb0e","Type":"ContainerDied","Data":"db33808df07b4ab5ff120bc8cf85057840450381a9874fc4c569f5f9b280ced0"}
Sep 17 14:06:09.399672 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:09.399269    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a65g8q" event={"ID":"ce73b866-39ce-41e2-b34f-831fcee4fb0e","Type":"ContainerStarted","Data":"6056ab8413ebb54890db7bc90d3787220860e8aae088ce604450ed20b082f3b1"}
Sep 17 14:06:09.401014 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:09.400997    2560 generic.go:356] "Generic (PLEG): container finished" podID="100093d5-c861-4ba1-a956-6fb0cc5dcadd" containerID="a4e754af492b1983d7b00c59e242e6e5c2be5221848e95fe2dfd1baeeb18921e" exitCode=0
Sep 17 14:06:09.401107 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:09.401065    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zlfbp" event={"ID":"100093d5-c861-4ba1-a956-6fb0cc5dcadd","Type":"ContainerDied","Data":"a4e754af492b1983d7b00c59e242e6e5c2be5221848e95fe2dfd1baeeb18921e"}
Sep 17 14:06:09.401190 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:09.401119    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zlfbp" event={"ID":"100093d5-c861-4ba1-a956-6fb0cc5dcadd","Type":"ContainerStarted","Data":"31f213ccf973dd0b299fd030baa1dc073e916dcbd13540eb80e628ac49214149"}
Sep 17 14:06:09.402616 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:09.402596    2560 generic.go:356] "Generic (PLEG): container finished" podID="6e5ee2d3-a52a-499b-a1d8-b707f3c6676a" containerID="e2cccb52b9ca2d823eef1af494a53207262f9328fdaac77c20793774466335aa" exitCode=0
Sep 17 14:06:09.402706 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:09.402681    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfhbgc" event={"ID":"6e5ee2d3-a52a-499b-a1d8-b707f3c6676a","Type":"ContainerDied","Data":"e2cccb52b9ca2d823eef1af494a53207262f9328fdaac77c20793774466335aa"}
Sep 17 14:06:09.402807 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:09.402720    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfhbgc" event={"ID":"6e5ee2d3-a52a-499b-a1d8-b707f3c6676a","Type":"ContainerStarted","Data":"ea790eacc4813eaa7dd0e6e6808887cbe7b3ee0e87c0e8079bdaec52dec8e015"}
Sep 17 14:06:09.404581 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:09.404559    2560 generic.go:356] "Generic (PLEG): container finished" podID="eee1c619-1676-4c44-b9d6-c01ae83c344c" containerID="7cf2125a7bd3ca2bb1317d966511160783f95af5ea6d90759ce2e505a7b45764" exitCode=0
Sep 17 14:06:09.404713 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:09.404639    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6487kn" event={"ID":"eee1c619-1676-4c44-b9d6-c01ae83c344c","Type":"ContainerDied","Data":"7cf2125a7bd3ca2bb1317d966511160783f95af5ea6d90759ce2e505a7b45764"}
Sep 17 14:06:09.404713 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:09.404683    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6487kn" event={"ID":"eee1c619-1676-4c44-b9d6-c01ae83c344c","Type":"ContainerStarted","Data":"21d5ece5c91db1fdd1b117d023d06b5277ec08fa407c43d185dcac78222c9996"}
Sep 17 14:06:10.412562 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:10.412534    2560 generic.go:356] "Generic (PLEG): container finished" podID="ce73b866-39ce-41e2-b34f-831fcee4fb0e" containerID="5b6e82e7f1550bc0959e07bc630e6b23aab648fe3a4a67d0154b9088b480d8f0" exitCode=0
Sep 17 14:06:10.412912 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:10.412606    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a65g8q" event={"ID":"ce73b866-39ce-41e2-b34f-831fcee4fb0e","Type":"ContainerDied","Data":"5b6e82e7f1550bc0959e07bc630e6b23aab648fe3a4a67d0154b9088b480d8f0"}
Sep 17 14:06:10.414624 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:10.414602    2560 generic.go:356] "Generic (PLEG): container finished" podID="100093d5-c861-4ba1-a956-6fb0cc5dcadd" containerID="f6eb5aa5fb71d8ae5301c04af7b05ed4c4107bdb6ce82cbc842ec0c72de37a10" exitCode=0
Sep 17 14:06:10.414719 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:10.414681    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zlfbp" event={"ID":"100093d5-c861-4ba1-a956-6fb0cc5dcadd","Type":"ContainerDied","Data":"f6eb5aa5fb71d8ae5301c04af7b05ed4c4107bdb6ce82cbc842ec0c72de37a10"}
Sep 17 14:06:10.416678 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:10.416652    2560 generic.go:356] "Generic (PLEG): container finished" podID="6e5ee2d3-a52a-499b-a1d8-b707f3c6676a" containerID="6ce497ac1262df0c594531c88ff80f0f811fd0edb59ac0f4ae5deb51783729b5" exitCode=0
Sep 17 14:06:10.416764 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:10.416733    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfhbgc" event={"ID":"6e5ee2d3-a52a-499b-a1d8-b707f3c6676a","Type":"ContainerDied","Data":"6ce497ac1262df0c594531c88ff80f0f811fd0edb59ac0f4ae5deb51783729b5"}
Sep 17 14:06:10.418877 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:10.418855    2560 generic.go:356] "Generic (PLEG): container finished" podID="eee1c619-1676-4c44-b9d6-c01ae83c344c" containerID="6204f72f184311ead34875ee6698f7e67a9a942cfe2b583730b1c9368b735059" exitCode=0
Sep 17 14:06:10.418963 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:10.418893    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6487kn" event={"ID":"eee1c619-1676-4c44-b9d6-c01ae83c344c","Type":"ContainerDied","Data":"6204f72f184311ead34875ee6698f7e67a9a942cfe2b583730b1c9368b735059"}
Sep 17 14:06:11.426149 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:11.426114    2560 generic.go:356] "Generic (PLEG): container finished" podID="eee1c619-1676-4c44-b9d6-c01ae83c344c" containerID="ed0173942e265058d588e9da4196edd10ce76d6b9ed568cc6129c2675186f03b" exitCode=0
Sep 17 14:06:11.426606 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:11.426210    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6487kn" event={"ID":"eee1c619-1676-4c44-b9d6-c01ae83c344c","Type":"ContainerDied","Data":"ed0173942e265058d588e9da4196edd10ce76d6b9ed568cc6129c2675186f03b"}
Sep 17 14:06:11.428232 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:11.428212    2560 generic.go:356] "Generic (PLEG): container finished" podID="ce73b866-39ce-41e2-b34f-831fcee4fb0e" containerID="4a3f44b3ccda7e2eac542fdae2699054a65f2bc11c2cff1bd40d564b7e5d201f" exitCode=0
Sep 17 14:06:11.428290 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:11.428279    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a65g8q" event={"ID":"ce73b866-39ce-41e2-b34f-831fcee4fb0e","Type":"ContainerDied","Data":"4a3f44b3ccda7e2eac542fdae2699054a65f2bc11c2cff1bd40d564b7e5d201f"}
Sep 17 14:06:11.430278 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:11.430254    2560 generic.go:356] "Generic (PLEG): container finished" podID="100093d5-c861-4ba1-a956-6fb0cc5dcadd" containerID="942d867076ea9799705b1de078788acfa04709c9c60416b01cdd40428ebe06e5" exitCode=0
Sep 17 14:06:11.430381 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:11.430340    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zlfbp" event={"ID":"100093d5-c861-4ba1-a956-6fb0cc5dcadd","Type":"ContainerDied","Data":"942d867076ea9799705b1de078788acfa04709c9c60416b01cdd40428ebe06e5"}
Sep 17 14:06:11.432318 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:11.432297    2560 generic.go:356] "Generic (PLEG): container finished" podID="6e5ee2d3-a52a-499b-a1d8-b707f3c6676a" containerID="99ab2e1535ba0d3f28ac13d9aaab567de2cc22a7aa9c9f4ff1ed6acb2bb34ca0" exitCode=0
Sep 17 14:06:11.432409 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:11.432373    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfhbgc" event={"ID":"6e5ee2d3-a52a-499b-a1d8-b707f3c6676a","Type":"ContainerDied","Data":"99ab2e1535ba0d3f28ac13d9aaab567de2cc22a7aa9c9f4ff1ed6acb2bb34ca0"}
Sep 17 14:06:12.586235 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:12.586207    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zlfbp"
Sep 17 14:06:12.636668 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:12.636642    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfhbgc"
Sep 17 14:06:12.639598 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:12.639574    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a65g8q"
Sep 17 14:06:12.665425 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:12.665400    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6487kn"
Sep 17 14:06:12.670495 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:12.670474    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wdthh\" (UniqueName: \"kubernetes.io/projected/100093d5-c861-4ba1-a956-6fb0cc5dcadd-kube-api-access-wdthh\") pod \"100093d5-c861-4ba1-a956-6fb0cc5dcadd\" (UID: \"100093d5-c861-4ba1-a956-6fb0cc5dcadd\") "
Sep 17 14:06:12.670603 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:12.670526    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/100093d5-c861-4ba1-a956-6fb0cc5dcadd-util\") pod \"100093d5-c861-4ba1-a956-6fb0cc5dcadd\" (UID: \"100093d5-c861-4ba1-a956-6fb0cc5dcadd\") "
Sep 17 14:06:12.670603 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:12.670588    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/100093d5-c861-4ba1-a956-6fb0cc5dcadd-bundle\") pod \"100093d5-c861-4ba1-a956-6fb0cc5dcadd\" (UID: \"100093d5-c861-4ba1-a956-6fb0cc5dcadd\") "
Sep 17 14:06:12.671101 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:12.671076    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/100093d5-c861-4ba1-a956-6fb0cc5dcadd-bundle" (OuterVolumeSpecName: "bundle") pod "100093d5-c861-4ba1-a956-6fb0cc5dcadd" (UID: "100093d5-c861-4ba1-a956-6fb0cc5dcadd"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:06:12.672536 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:12.672513    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/100093d5-c861-4ba1-a956-6fb0cc5dcadd-kube-api-access-wdthh" (OuterVolumeSpecName: "kube-api-access-wdthh") pod "100093d5-c861-4ba1-a956-6fb0cc5dcadd" (UID: "100093d5-c861-4ba1-a956-6fb0cc5dcadd"). InnerVolumeSpecName "kube-api-access-wdthh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:06:12.676032 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:12.676011    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/100093d5-c861-4ba1-a956-6fb0cc5dcadd-util" (OuterVolumeSpecName: "util") pod "100093d5-c861-4ba1-a956-6fb0cc5dcadd" (UID: "100093d5-c861-4ba1-a956-6fb0cc5dcadd"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:06:12.771702 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:12.771614    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-psxl6\" (UniqueName: \"kubernetes.io/projected/6e5ee2d3-a52a-499b-a1d8-b707f3c6676a-kube-api-access-psxl6\") pod \"6e5ee2d3-a52a-499b-a1d8-b707f3c6676a\" (UID: \"6e5ee2d3-a52a-499b-a1d8-b707f3c6676a\") "
Sep 17 14:06:12.771702 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:12.771676    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6e5ee2d3-a52a-499b-a1d8-b707f3c6676a-util\") pod \"6e5ee2d3-a52a-499b-a1d8-b707f3c6676a\" (UID: \"6e5ee2d3-a52a-499b-a1d8-b707f3c6676a\") "
Sep 17 14:06:12.771702 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:12.771697    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6e5ee2d3-a52a-499b-a1d8-b707f3c6676a-bundle\") pod \"6e5ee2d3-a52a-499b-a1d8-b707f3c6676a\" (UID: \"6e5ee2d3-a52a-499b-a1d8-b707f3c6676a\") "
Sep 17 14:06:12.771972 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:12.771742    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lnckq\" (UniqueName: \"kubernetes.io/projected/ce73b866-39ce-41e2-b34f-831fcee4fb0e-kube-api-access-lnckq\") pod \"ce73b866-39ce-41e2-b34f-831fcee4fb0e\" (UID: \"ce73b866-39ce-41e2-b34f-831fcee4fb0e\") "
Sep 17 14:06:12.771972 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:12.771795    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ce73b866-39ce-41e2-b34f-831fcee4fb0e-util\") pod \"ce73b866-39ce-41e2-b34f-831fcee4fb0e\" (UID: \"ce73b866-39ce-41e2-b34f-831fcee4fb0e\") "
Sep 17 14:06:12.771972 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:12.771862    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eee1c619-1676-4c44-b9d6-c01ae83c344c-bundle\") pod \"eee1c619-1676-4c44-b9d6-c01ae83c344c\" (UID: \"eee1c619-1676-4c44-b9d6-c01ae83c344c\") "
Sep 17 14:06:12.771972 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:12.771886    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r94tw\" (UniqueName: \"kubernetes.io/projected/eee1c619-1676-4c44-b9d6-c01ae83c344c-kube-api-access-r94tw\") pod \"eee1c619-1676-4c44-b9d6-c01ae83c344c\" (UID: \"eee1c619-1676-4c44-b9d6-c01ae83c344c\") "
Sep 17 14:06:12.771972 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:12.771911    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce73b866-39ce-41e2-b34f-831fcee4fb0e-bundle\") pod \"ce73b866-39ce-41e2-b34f-831fcee4fb0e\" (UID: \"ce73b866-39ce-41e2-b34f-831fcee4fb0e\") "
Sep 17 14:06:12.771972 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:12.771939    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eee1c619-1676-4c44-b9d6-c01ae83c344c-util\") pod \"eee1c619-1676-4c44-b9d6-c01ae83c344c\" (UID: \"eee1c619-1676-4c44-b9d6-c01ae83c344c\") "
Sep 17 14:06:12.772278 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:12.772252    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdthh\" (UniqueName: \"kubernetes.io/projected/100093d5-c861-4ba1-a956-6fb0cc5dcadd-kube-api-access-wdthh\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:06:12.772278 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:12.772271    2560 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/100093d5-c861-4ba1-a956-6fb0cc5dcadd-util\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:06:12.772391 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:12.772285    2560 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/100093d5-c861-4ba1-a956-6fb0cc5dcadd-bundle\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:06:12.772391 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:12.772304    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e5ee2d3-a52a-499b-a1d8-b707f3c6676a-bundle" (OuterVolumeSpecName: "bundle") pod "6e5ee2d3-a52a-499b-a1d8-b707f3c6676a" (UID: "6e5ee2d3-a52a-499b-a1d8-b707f3c6676a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:06:12.773380 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:12.772862    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eee1c619-1676-4c44-b9d6-c01ae83c344c-bundle" (OuterVolumeSpecName: "bundle") pod "eee1c619-1676-4c44-b9d6-c01ae83c344c" (UID: "eee1c619-1676-4c44-b9d6-c01ae83c344c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:06:12.773380 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:12.773336    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce73b866-39ce-41e2-b34f-831fcee4fb0e-bundle" (OuterVolumeSpecName: "bundle") pod "ce73b866-39ce-41e2-b34f-831fcee4fb0e" (UID: "ce73b866-39ce-41e2-b34f-831fcee4fb0e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:06:12.774128 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:12.774103    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e5ee2d3-a52a-499b-a1d8-b707f3c6676a-kube-api-access-psxl6" (OuterVolumeSpecName: "kube-api-access-psxl6") pod "6e5ee2d3-a52a-499b-a1d8-b707f3c6676a" (UID: "6e5ee2d3-a52a-499b-a1d8-b707f3c6676a"). InnerVolumeSpecName "kube-api-access-psxl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:06:12.774407 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:12.774385    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eee1c619-1676-4c44-b9d6-c01ae83c344c-kube-api-access-r94tw" (OuterVolumeSpecName: "kube-api-access-r94tw") pod "eee1c619-1676-4c44-b9d6-c01ae83c344c" (UID: "eee1c619-1676-4c44-b9d6-c01ae83c344c"). InnerVolumeSpecName "kube-api-access-r94tw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:06:12.774868 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:12.774840    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce73b866-39ce-41e2-b34f-831fcee4fb0e-kube-api-access-lnckq" (OuterVolumeSpecName: "kube-api-access-lnckq") pod "ce73b866-39ce-41e2-b34f-831fcee4fb0e" (UID: "ce73b866-39ce-41e2-b34f-831fcee4fb0e"). InnerVolumeSpecName "kube-api-access-lnckq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:06:12.778501 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:12.778480    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eee1c619-1676-4c44-b9d6-c01ae83c344c-util" (OuterVolumeSpecName: "util") pod "eee1c619-1676-4c44-b9d6-c01ae83c344c" (UID: "eee1c619-1676-4c44-b9d6-c01ae83c344c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:06:12.780762 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:12.780737    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e5ee2d3-a52a-499b-a1d8-b707f3c6676a-util" (OuterVolumeSpecName: "util") pod "6e5ee2d3-a52a-499b-a1d8-b707f3c6676a" (UID: "6e5ee2d3-a52a-499b-a1d8-b707f3c6676a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:06:12.780920 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:12.780903    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce73b866-39ce-41e2-b34f-831fcee4fb0e-util" (OuterVolumeSpecName: "util") pod "ce73b866-39ce-41e2-b34f-831fcee4fb0e" (UID: "ce73b866-39ce-41e2-b34f-831fcee4fb0e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:06:12.873498 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:12.873444    2560 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6e5ee2d3-a52a-499b-a1d8-b707f3c6676a-util\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:06:12.873498 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:12.873495    2560 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6e5ee2d3-a52a-499b-a1d8-b707f3c6676a-bundle\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:06:12.873665 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:12.873506    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnckq\" (UniqueName: \"kubernetes.io/projected/ce73b866-39ce-41e2-b34f-831fcee4fb0e-kube-api-access-lnckq\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:06:12.873665 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:12.873516    2560 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ce73b866-39ce-41e2-b34f-831fcee4fb0e-util\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:06:12.873665 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:12.873524    2560 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eee1c619-1676-4c44-b9d6-c01ae83c344c-bundle\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:06:12.873665 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:12.873533    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r94tw\" (UniqueName: \"kubernetes.io/projected/eee1c619-1676-4c44-b9d6-c01ae83c344c-kube-api-access-r94tw\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:06:12.873665 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:12.873540    2560 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce73b866-39ce-41e2-b34f-831fcee4fb0e-bundle\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:06:12.873665 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:12.873549    2560 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eee1c619-1676-4c44-b9d6-c01ae83c344c-util\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:06:12.873665 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:12.873556    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psxl6\" (UniqueName: \"kubernetes.io/projected/6e5ee2d3-a52a-499b-a1d8-b707f3c6676a-kube-api-access-psxl6\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:06:13.445650 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:13.445623    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a65g8q"
Sep 17 14:06:13.445838 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:13.445617    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a65g8q" event={"ID":"ce73b866-39ce-41e2-b34f-831fcee4fb0e","Type":"ContainerDied","Data":"6056ab8413ebb54890db7bc90d3787220860e8aae088ce604450ed20b082f3b1"}
Sep 17 14:06:13.445838 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:13.445777    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6056ab8413ebb54890db7bc90d3787220860e8aae088ce604450ed20b082f3b1"
Sep 17 14:06:13.447603 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:13.447581    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zlfbp" event={"ID":"100093d5-c861-4ba1-a956-6fb0cc5dcadd","Type":"ContainerDied","Data":"31f213ccf973dd0b299fd030baa1dc073e916dcbd13540eb80e628ac49214149"}
Sep 17 14:06:13.447603 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:13.447602    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zlfbp"
Sep 17 14:06:13.447755 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:13.447614    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31f213ccf973dd0b299fd030baa1dc073e916dcbd13540eb80e628ac49214149"
Sep 17 14:06:13.449495 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:13.449478    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfhbgc"
Sep 17 14:06:13.449605 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:13.449495    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfhbgc" event={"ID":"6e5ee2d3-a52a-499b-a1d8-b707f3c6676a","Type":"ContainerDied","Data":"ea790eacc4813eaa7dd0e6e6808887cbe7b3ee0e87c0e8079bdaec52dec8e015"}
Sep 17 14:06:13.449605 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:13.449536    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea790eacc4813eaa7dd0e6e6808887cbe7b3ee0e87c0e8079bdaec52dec8e015"
Sep 17 14:06:13.451415 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:13.451398    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6487kn" event={"ID":"eee1c619-1676-4c44-b9d6-c01ae83c344c","Type":"ContainerDied","Data":"21d5ece5c91db1fdd1b117d023d06b5277ec08fa407c43d185dcac78222c9996"}
Sep 17 14:06:13.451522 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:13.451430    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21d5ece5c91db1fdd1b117d023d06b5277ec08fa407c43d185dcac78222c9996"
Sep 17 14:06:13.451522 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:13.451439    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6487kn"
Sep 17 14:06:19.984189 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:19.984146    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:06:20.801431 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:20.801400    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:06:23.124567 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:23.124534    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:06:23.810206 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:23.810170    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:06:24.491825 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:24.491791    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:06:26.198873 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:26.198842    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:06:26.599819 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:26.599782    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:06:27.015209 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:27.015129    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:06:27.897848 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:27.897816    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:06:28.009254 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:28.009218    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:06:30.423774 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:30.423734    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:06:30.521496 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:30.521449    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:06:30.815942 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:30.815899    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:06:33.227629 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:33.227594    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:06:33.345731 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:33.345694    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:06:33.484826 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:33.484729    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:06:33.582774 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:33.582737    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:06:37.028042 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:37.028006    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:06:37.141658 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:37.141626    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:06:37.302822 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:37.302782    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:06:37.499418 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:37.499383    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:06:40.264390 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:40.264354    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:06:40.365583 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:40.365544    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:06:42.768962 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:42.768932    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:06:42.868985 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:42.868954    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:06:42.972531 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:42.972492    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:06:49.895668 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:49.895631    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:06:50.009873 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:50.009833    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:06:53.009363 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:53.009328    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:06:53.110268 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:53.110237    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:06:56.473986 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:56.473952    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:06:56.543643 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:06:56.543609    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:07:02.313427 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:02.313392    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:07:02.440655 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:02.440625    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:07:02.526515 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:02.526479    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:07:08.671002 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:08.670974    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:07:09.385729 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:09.385692    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:07:09.585396 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:09.585366    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:07:15.819039 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:15.819006    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:07:17.522091 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:17.522053    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:07:20.765836 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:20.765795    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:07:20.864511 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:20.864481    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:07:21.012127 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:21.012101    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:07:21.631582 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:21.631551    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-dk5vs"]
Sep 17 14:07:21.632131 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:21.632111    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce73b866-39ce-41e2-b34f-831fcee4fb0e" containerName="extract"
Sep 17 14:07:21.632197 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:21.632136    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce73b866-39ce-41e2-b34f-831fcee4fb0e" containerName="extract"
Sep 17 14:07:21.632197 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:21.632150    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e5ee2d3-a52a-499b-a1d8-b707f3c6676a" containerName="extract"
Sep 17 14:07:21.632197 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:21.632160    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e5ee2d3-a52a-499b-a1d8-b707f3c6676a" containerName="extract"
Sep 17 14:07:21.632197 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:21.632172    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="100093d5-c861-4ba1-a956-6fb0cc5dcadd" containerName="util"
Sep 17 14:07:21.632197 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:21.632187    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="100093d5-c861-4ba1-a956-6fb0cc5dcadd" containerName="util"
Sep 17 14:07:21.632356 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:21.632202    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce73b866-39ce-41e2-b34f-831fcee4fb0e" containerName="pull"
Sep 17 14:07:21.632356 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:21.632210    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce73b866-39ce-41e2-b34f-831fcee4fb0e" containerName="pull"
Sep 17 14:07:21.632356 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:21.632227    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eee1c619-1676-4c44-b9d6-c01ae83c344c" containerName="pull"
Sep 17 14:07:21.632356 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:21.632235    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="eee1c619-1676-4c44-b9d6-c01ae83c344c" containerName="pull"
Sep 17 14:07:21.632356 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:21.632248    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eee1c619-1676-4c44-b9d6-c01ae83c344c" containerName="extract"
Sep 17 14:07:21.632356 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:21.632256    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="eee1c619-1676-4c44-b9d6-c01ae83c344c" containerName="extract"
Sep 17 14:07:21.632356 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:21.632276    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eee1c619-1676-4c44-b9d6-c01ae83c344c" containerName="util"
Sep 17 14:07:21.632356 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:21.632284    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="eee1c619-1676-4c44-b9d6-c01ae83c344c" containerName="util"
Sep 17 14:07:21.632356 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:21.632295    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="100093d5-c861-4ba1-a956-6fb0cc5dcadd" containerName="extract"
Sep 17 14:07:21.632356 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:21.632302    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="100093d5-c861-4ba1-a956-6fb0cc5dcadd" containerName="extract"
Sep 17 14:07:21.632356 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:21.632315    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce73b866-39ce-41e2-b34f-831fcee4fb0e" containerName="util"
Sep 17 14:07:21.632356 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:21.632322    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce73b866-39ce-41e2-b34f-831fcee4fb0e" containerName="util"
Sep 17 14:07:21.632356 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:21.632339    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e5ee2d3-a52a-499b-a1d8-b707f3c6676a" containerName="util"
Sep 17 14:07:21.632356 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:21.632347    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e5ee2d3-a52a-499b-a1d8-b707f3c6676a" containerName="util"
Sep 17 14:07:21.632356 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:21.632355    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="100093d5-c861-4ba1-a956-6fb0cc5dcadd" containerName="pull"
Sep 17 14:07:21.633042 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:21.632363    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="100093d5-c861-4ba1-a956-6fb0cc5dcadd" containerName="pull"
Sep 17 14:07:21.633042 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:21.632373    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e5ee2d3-a52a-499b-a1d8-b707f3c6676a" containerName="pull"
Sep 17 14:07:21.633042 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:21.632381    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e5ee2d3-a52a-499b-a1d8-b707f3c6676a" containerName="pull"
Sep 17 14:07:21.633042 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:21.632490    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="100093d5-c861-4ba1-a956-6fb0cc5dcadd" containerName="extract"
Sep 17 14:07:21.633042 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:21.632504    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="eee1c619-1676-4c44-b9d6-c01ae83c344c" containerName="extract"
Sep 17 14:07:21.633042 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:21.632513    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e5ee2d3-a52a-499b-a1d8-b707f3c6676a" containerName="extract"
Sep 17 14:07:21.633042 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:21.632519    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce73b866-39ce-41e2-b34f-831fcee4fb0e" containerName="extract"
Sep 17 14:07:21.635698 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:21.635677    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-dk5vs"
Sep 17 14:07:21.641214 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:21.641197    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 17 14:07:21.641767 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:21.641750    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-tsglc\""
Sep 17 14:07:21.642971 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:21.642952    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 17 14:07:21.650143 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:21.650122    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-dk5vs"]
Sep 17 14:07:21.698940 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:21.698911    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:07:21.756645 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:21.756617    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4gml\" (UniqueName: \"kubernetes.io/projected/05757844-b628-45c8-aca8-4802f7b0f42a-kube-api-access-p4gml\") pod \"authorino-7bcfd8f9cd-dk5vs\" (UID: \"05757844-b628-45c8-aca8-4802f7b0f42a\") " pod="kuadrant-system/authorino-7bcfd8f9cd-dk5vs"
Sep 17 14:07:21.804397 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:21.804363    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:07:21.858142 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:21.858111    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4gml\" (UniqueName: \"kubernetes.io/projected/05757844-b628-45c8-aca8-4802f7b0f42a-kube-api-access-p4gml\") pod \"authorino-7bcfd8f9cd-dk5vs\" (UID: \"05757844-b628-45c8-aca8-4802f7b0f42a\") " pod="kuadrant-system/authorino-7bcfd8f9cd-dk5vs"
Sep 17 14:07:21.868344 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:21.868316    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4gml\" (UniqueName: \"kubernetes.io/projected/05757844-b628-45c8-aca8-4802f7b0f42a-kube-api-access-p4gml\") pod \"authorino-7bcfd8f9cd-dk5vs\" (UID: \"05757844-b628-45c8-aca8-4802f7b0f42a\") " pod="kuadrant-system/authorino-7bcfd8f9cd-dk5vs"
Sep 17 14:07:21.946009 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:21.945953    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-dk5vs"
Sep 17 14:07:22.226471 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:22.226437    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-dk5vs"]
Sep 17 14:07:22.228789 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:07:22.228748    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod05757844_b628_45c8_aca8_4802f7b0f42a.slice/crio-7ca0e1e834b38028c8181777bf0b534e47a5448add3e21c20248111ddecdf054 WatchSource:0}: Error finding container 7ca0e1e834b38028c8181777bf0b534e47a5448add3e21c20248111ddecdf054: Status 404 returned error can't find the container with id 7ca0e1e834b38028c8181777bf0b534e47a5448add3e21c20248111ddecdf054
Sep 17 14:07:22.822990 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:22.822958    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-dk5vs" event={"ID":"05757844-b628-45c8-aca8-4802f7b0f42a","Type":"ContainerStarted","Data":"7ca0e1e834b38028c8181777bf0b534e47a5448add3e21c20248111ddecdf054"}
Sep 17 14:07:24.837881 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:24.837828    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-dk5vs" event={"ID":"05757844-b628-45c8-aca8-4802f7b0f42a","Type":"ContainerStarted","Data":"13ba945d56c821cabd400631847fb79e8d615ea020fdcf174bfd089eac6b9758"}
Sep 17 14:07:24.852632 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:24.852579    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-dk5vs" podStartSLOduration=1.439320263 podStartE2EDuration="3.852563581s" totalImagesPullingTime="2.413243318s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:07:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:07:24.85215844 +0000 UTC m=+713.794302098" watchObservedRunningTime="2026-09-17 14:07:24.852563581 +0000 UTC m=+713.794707274"
Sep 17 14:07:25.840232 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:25.840200    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-dk5vs"]
Sep 17 14:07:26.849682 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:26.849647    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-dk5vs" podUID="05757844-b628-45c8-aca8-4802f7b0f42a" containerName="authorino" containerID="cri-o://13ba945d56c821cabd400631847fb79e8d615ea020fdcf174bfd089eac6b9758" gracePeriod=30
Sep 17 14:07:27.094717 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:27.094695    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-dk5vs"
Sep 17 14:07:27.206206 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:27.206131    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p4gml\" (UniqueName: \"kubernetes.io/projected/05757844-b628-45c8-aca8-4802f7b0f42a-kube-api-access-p4gml\") pod \"05757844-b628-45c8-aca8-4802f7b0f42a\" (UID: \"05757844-b628-45c8-aca8-4802f7b0f42a\") "
Sep 17 14:07:27.208209 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:27.208172    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05757844-b628-45c8-aca8-4802f7b0f42a-kube-api-access-p4gml" (OuterVolumeSpecName: "kube-api-access-p4gml") pod "05757844-b628-45c8-aca8-4802f7b0f42a" (UID: "05757844-b628-45c8-aca8-4802f7b0f42a"). InnerVolumeSpecName "kube-api-access-p4gml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:07:27.307163 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:27.307139    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4gml\" (UniqueName: \"kubernetes.io/projected/05757844-b628-45c8-aca8-4802f7b0f42a-kube-api-access-p4gml\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:07:27.856375 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:27.856285    2560 generic.go:356] "Generic (PLEG): container finished" podID="05757844-b628-45c8-aca8-4802f7b0f42a" containerID="13ba945d56c821cabd400631847fb79e8d615ea020fdcf174bfd089eac6b9758" exitCode=0
Sep 17 14:07:27.856375 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:27.856331    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-dk5vs"
Sep 17 14:07:27.856899 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:27.856369    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-dk5vs" event={"ID":"05757844-b628-45c8-aca8-4802f7b0f42a","Type":"ContainerDied","Data":"13ba945d56c821cabd400631847fb79e8d615ea020fdcf174bfd089eac6b9758"}
Sep 17 14:07:27.856899 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:27.856411    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-dk5vs" event={"ID":"05757844-b628-45c8-aca8-4802f7b0f42a","Type":"ContainerDied","Data":"7ca0e1e834b38028c8181777bf0b534e47a5448add3e21c20248111ddecdf054"}
Sep 17 14:07:27.856899 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:27.856428    2560 scope.go:117] "RemoveContainer" containerID="13ba945d56c821cabd400631847fb79e8d615ea020fdcf174bfd089eac6b9758"
Sep 17 14:07:27.865069 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:27.865053    2560 scope.go:117] "RemoveContainer" containerID="13ba945d56c821cabd400631847fb79e8d615ea020fdcf174bfd089eac6b9758"
Sep 17 14:07:27.865320 ip-10-0-129-158 kubenswrapper[2560]: E0917 14:07:27.865302    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13ba945d56c821cabd400631847fb79e8d615ea020fdcf174bfd089eac6b9758\": container with ID starting with 13ba945d56c821cabd400631847fb79e8d615ea020fdcf174bfd089eac6b9758 not found: ID does not exist" containerID="13ba945d56c821cabd400631847fb79e8d615ea020fdcf174bfd089eac6b9758"
Sep 17 14:07:27.865370 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:27.865327    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13ba945d56c821cabd400631847fb79e8d615ea020fdcf174bfd089eac6b9758"} err="failed to get container status \"13ba945d56c821cabd400631847fb79e8d615ea020fdcf174bfd089eac6b9758\": rpc error: code = NotFound desc = could not find container \"13ba945d56c821cabd400631847fb79e8d615ea020fdcf174bfd089eac6b9758\": container with ID starting with 13ba945d56c821cabd400631847fb79e8d615ea020fdcf174bfd089eac6b9758 not found: ID does not exist"
Sep 17 14:07:27.874992 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:27.874972    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-dk5vs"]
Sep 17 14:07:27.879206 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:27.879187    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-dk5vs"]
Sep 17 14:07:29.569055 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:29.569015    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05757844-b628-45c8-aca8-4802f7b0f42a" path="/var/lib/kubelet/pods/05757844-b628-45c8-aca8-4802f7b0f42a/volumes"
Sep 17 14:07:42.019169 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:42.019126    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:07:42.131659 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:42.131628    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:07:42.493240 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:42.493186    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:07:54.825262 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:54.825217    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:07:55.017410 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:55.017378    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-ltqwh"]
Sep 17 14:07:55.017799 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:55.017784    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05757844-b628-45c8-aca8-4802f7b0f42a" containerName="authorino"
Sep 17 14:07:55.017845 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:55.017801    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="05757844-b628-45c8-aca8-4802f7b0f42a" containerName="authorino"
Sep 17 14:07:55.017894 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:55.017882    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="05757844-b628-45c8-aca8-4802f7b0f42a" containerName="authorino"
Sep 17 14:07:55.022129 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:55.022112    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-ltqwh"
Sep 17 14:07:55.024367 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:55.024342    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 17 14:07:55.025123 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:55.025107    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 17 14:07:55.025207 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:55.025148    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-tsglc\""
Sep 17 14:07:55.029038 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:55.029017    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-ltqwh"]
Sep 17 14:07:55.129600 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:55.129517    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvkqt\" (UniqueName: \"kubernetes.io/projected/94acd192-6343-431e-8711-f16dc6ef34d9-kube-api-access-dvkqt\") pod \"authorino-557f876f69-ltqwh\" (UID: \"94acd192-6343-431e-8711-f16dc6ef34d9\") " pod="kuadrant-system/authorino-557f876f69-ltqwh"
Sep 17 14:07:55.230986 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:55.230955    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvkqt\" (UniqueName: \"kubernetes.io/projected/94acd192-6343-431e-8711-f16dc6ef34d9-kube-api-access-dvkqt\") pod \"authorino-557f876f69-ltqwh\" (UID: \"94acd192-6343-431e-8711-f16dc6ef34d9\") " pod="kuadrant-system/authorino-557f876f69-ltqwh"
Sep 17 14:07:55.239840 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:55.239816    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvkqt\" (UniqueName: \"kubernetes.io/projected/94acd192-6343-431e-8711-f16dc6ef34d9-kube-api-access-dvkqt\") pod \"authorino-557f876f69-ltqwh\" (UID: \"94acd192-6343-431e-8711-f16dc6ef34d9\") " pod="kuadrant-system/authorino-557f876f69-ltqwh"
Sep 17 14:07:55.251554 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:55.251518    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-ltqwh"]
Sep 17 14:07:55.251756 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:55.251743    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-ltqwh"
Sep 17 14:07:55.280343 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:55.280318    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-749d46758f-gz2cr"]
Sep 17 14:07:55.283934 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:55.283918    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-749d46758f-gz2cr"
Sep 17 14:07:55.294878 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:55.294855    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-749d46758f-gz2cr"]
Sep 17 14:07:55.332245 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:55.332207    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk74m\" (UniqueName: \"kubernetes.io/projected/47218685-e47f-4165-89e6-8dfc16504c60-kube-api-access-gk74m\") pod \"authorino-749d46758f-gz2cr\" (UID: \"47218685-e47f-4165-89e6-8dfc16504c60\") " pod="kuadrant-system/authorino-749d46758f-gz2cr"
Sep 17 14:07:55.407477 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:55.407390    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:07:55.433700 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:55.433675    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk74m\" (UniqueName: \"kubernetes.io/projected/47218685-e47f-4165-89e6-8dfc16504c60-kube-api-access-gk74m\") pod \"authorino-749d46758f-gz2cr\" (UID: \"47218685-e47f-4165-89e6-8dfc16504c60\") " pod="kuadrant-system/authorino-749d46758f-gz2cr"
Sep 17 14:07:55.441452 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:55.441433    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk74m\" (UniqueName: \"kubernetes.io/projected/47218685-e47f-4165-89e6-8dfc16504c60-kube-api-access-gk74m\") pod \"authorino-749d46758f-gz2cr\" (UID: \"47218685-e47f-4165-89e6-8dfc16504c60\") " pod="kuadrant-system/authorino-749d46758f-gz2cr"
Sep 17 14:07:55.525579 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:55.525555    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-ltqwh"]
Sep 17 14:07:55.528063 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:07:55.528036    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod94acd192_6343_431e_8711_f16dc6ef34d9.slice/crio-ac4dc2c3164ce475749c8d8eb05bb7565b97350042234adabcd0023505393ca1 WatchSource:0}: Error finding container ac4dc2c3164ce475749c8d8eb05bb7565b97350042234adabcd0023505393ca1: Status 404 returned error can't find the container with id ac4dc2c3164ce475749c8d8eb05bb7565b97350042234adabcd0023505393ca1
Sep 17 14:07:55.595304 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:55.595277    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-749d46758f-gz2cr"
Sep 17 14:07:55.867825 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:55.867800    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-749d46758f-gz2cr"]
Sep 17 14:07:55.869080 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:07:55.869054    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod47218685_e47f_4165_89e6_8dfc16504c60.slice/crio-e43dd36495a535866a8798e8a55e9ae3643986fbadd2947b157d5d08ce3cace8 WatchSource:0}: Error finding container e43dd36495a535866a8798e8a55e9ae3643986fbadd2947b157d5d08ce3cace8: Status 404 returned error can't find the container with id e43dd36495a535866a8798e8a55e9ae3643986fbadd2947b157d5d08ce3cace8
Sep 17 14:07:56.010355 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:56.010326    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-749d46758f-gz2cr" event={"ID":"47218685-e47f-4165-89e6-8dfc16504c60","Type":"ContainerStarted","Data":"e43dd36495a535866a8798e8a55e9ae3643986fbadd2947b157d5d08ce3cace8"}
Sep 17 14:07:56.011704 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:56.011682    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-ltqwh" event={"ID":"94acd192-6343-431e-8711-f16dc6ef34d9","Type":"ContainerStarted","Data":"ac4dc2c3164ce475749c8d8eb05bb7565b97350042234adabcd0023505393ca1"}
Sep 17 14:07:57.018851 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:57.018813    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-ltqwh" event={"ID":"94acd192-6343-431e-8711-f16dc6ef34d9","Type":"ContainerStarted","Data":"3e156ae889819ee4aaec0a6796c7dbb0ce5fffd501ba8ce73ee43b60891e9453"}
Sep 17 14:07:57.019273 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:57.018855    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-557f876f69-ltqwh" podUID="94acd192-6343-431e-8711-f16dc6ef34d9" containerName="authorino" containerID="cri-o://3e156ae889819ee4aaec0a6796c7dbb0ce5fffd501ba8ce73ee43b60891e9453" gracePeriod=30
Sep 17 14:07:57.020674 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:57.020651    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-749d46758f-gz2cr" event={"ID":"47218685-e47f-4165-89e6-8dfc16504c60","Type":"ContainerStarted","Data":"d37b3fc9c4ef789bbe66b6a4596cc182d2c209e4eecccc93465d94fc1cf1402d"}
Sep 17 14:07:57.035334 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:57.035295    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-557f876f69-ltqwh" podStartSLOduration=2.348719368 podStartE2EDuration="3.035280711s" totalImagesPullingTime="686.561343ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:07:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:07:57.033118561 +0000 UTC m=+745.975262218" watchObservedRunningTime="2026-09-17 14:07:57.035280711 +0000 UTC m=+745.977424357"
Sep 17 14:07:57.048546 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:57.048508    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-749d46758f-gz2cr" podStartSLOduration=1.6522989799999999 podStartE2EDuration="2.048498556s" totalImagesPullingTime="396.199576ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:07:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:07:57.047047504 +0000 UTC m=+745.989191150" watchObservedRunningTime="2026-09-17 14:07:57.048498556 +0000 UTC m=+745.990642202"
Sep 17 14:07:57.279527 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:57.279329    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-ltqwh"
Sep 17 14:07:57.348117 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:57.348091    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dvkqt\" (UniqueName: \"kubernetes.io/projected/94acd192-6343-431e-8711-f16dc6ef34d9-kube-api-access-dvkqt\") pod \"94acd192-6343-431e-8711-f16dc6ef34d9\" (UID: \"94acd192-6343-431e-8711-f16dc6ef34d9\") "
Sep 17 14:07:57.349950 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:57.349930    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94acd192-6343-431e-8711-f16dc6ef34d9-kube-api-access-dvkqt" (OuterVolumeSpecName: "kube-api-access-dvkqt") pod "94acd192-6343-431e-8711-f16dc6ef34d9" (UID: "94acd192-6343-431e-8711-f16dc6ef34d9"). InnerVolumeSpecName "kube-api-access-dvkqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:07:57.449662 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:57.449638    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvkqt\" (UniqueName: \"kubernetes.io/projected/94acd192-6343-431e-8711-f16dc6ef34d9-kube-api-access-dvkqt\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:07:58.027437 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:58.027358    2560 generic.go:356] "Generic (PLEG): container finished" podID="94acd192-6343-431e-8711-f16dc6ef34d9" containerID="3e156ae889819ee4aaec0a6796c7dbb0ce5fffd501ba8ce73ee43b60891e9453" exitCode=0
Sep 17 14:07:58.027437 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:58.027412    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-ltqwh"
Sep 17 14:07:58.027895 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:58.027452    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-ltqwh" event={"ID":"94acd192-6343-431e-8711-f16dc6ef34d9","Type":"ContainerDied","Data":"3e156ae889819ee4aaec0a6796c7dbb0ce5fffd501ba8ce73ee43b60891e9453"}
Sep 17 14:07:58.027895 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:58.027509    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-ltqwh" event={"ID":"94acd192-6343-431e-8711-f16dc6ef34d9","Type":"ContainerDied","Data":"ac4dc2c3164ce475749c8d8eb05bb7565b97350042234adabcd0023505393ca1"}
Sep 17 14:07:58.027895 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:58.027527    2560 scope.go:117] "RemoveContainer" containerID="3e156ae889819ee4aaec0a6796c7dbb0ce5fffd501ba8ce73ee43b60891e9453"
Sep 17 14:07:58.036261 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:58.036245    2560 scope.go:117] "RemoveContainer" containerID="3e156ae889819ee4aaec0a6796c7dbb0ce5fffd501ba8ce73ee43b60891e9453"
Sep 17 14:07:58.036532 ip-10-0-129-158 kubenswrapper[2560]: E0917 14:07:58.036514    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e156ae889819ee4aaec0a6796c7dbb0ce5fffd501ba8ce73ee43b60891e9453\": container with ID starting with 3e156ae889819ee4aaec0a6796c7dbb0ce5fffd501ba8ce73ee43b60891e9453 not found: ID does not exist" containerID="3e156ae889819ee4aaec0a6796c7dbb0ce5fffd501ba8ce73ee43b60891e9453"
Sep 17 14:07:58.036591 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:58.036539    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e156ae889819ee4aaec0a6796c7dbb0ce5fffd501ba8ce73ee43b60891e9453"} err="failed to get container status \"3e156ae889819ee4aaec0a6796c7dbb0ce5fffd501ba8ce73ee43b60891e9453\": rpc error: code = NotFound desc = could not find container \"3e156ae889819ee4aaec0a6796c7dbb0ce5fffd501ba8ce73ee43b60891e9453\": container with ID starting with 3e156ae889819ee4aaec0a6796c7dbb0ce5fffd501ba8ce73ee43b60891e9453 not found: ID does not exist"
Sep 17 14:07:58.046650 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:58.046628    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-ltqwh"]
Sep 17 14:07:58.050035 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:58.050011    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-ltqwh"]
Sep 17 14:07:58.990419 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:58.990388    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-749d46758f-gz2cr"]
Sep 17 14:07:59.034405 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:59.034375    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-749d46758f-gz2cr" podUID="47218685-e47f-4165-89e6-8dfc16504c60" containerName="authorino" containerID="cri-o://d37b3fc9c4ef789bbe66b6a4596cc182d2c209e4eecccc93465d94fc1cf1402d" gracePeriod=30
Sep 17 14:07:59.282139 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:59.282115    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-749d46758f-gz2cr"
Sep 17 14:07:59.368303 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:59.368264    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gk74m\" (UniqueName: \"kubernetes.io/projected/47218685-e47f-4165-89e6-8dfc16504c60-kube-api-access-gk74m\") pod \"47218685-e47f-4165-89e6-8dfc16504c60\" (UID: \"47218685-e47f-4165-89e6-8dfc16504c60\") "
Sep 17 14:07:59.370200 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:59.370178    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47218685-e47f-4165-89e6-8dfc16504c60-kube-api-access-gk74m" (OuterVolumeSpecName: "kube-api-access-gk74m") pod "47218685-e47f-4165-89e6-8dfc16504c60" (UID: "47218685-e47f-4165-89e6-8dfc16504c60"). InnerVolumeSpecName "kube-api-access-gk74m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:07:59.469608 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:59.469584    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gk74m\" (UniqueName: \"kubernetes.io/projected/47218685-e47f-4165-89e6-8dfc16504c60-kube-api-access-gk74m\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:07:59.569406 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:07:59.569378    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94acd192-6343-431e-8711-f16dc6ef34d9" path="/var/lib/kubelet/pods/94acd192-6343-431e-8711-f16dc6ef34d9/volumes"
Sep 17 14:08:00.040812 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:08:00.040726    2560 generic.go:356] "Generic (PLEG): container finished" podID="47218685-e47f-4165-89e6-8dfc16504c60" containerID="d37b3fc9c4ef789bbe66b6a4596cc182d2c209e4eecccc93465d94fc1cf1402d" exitCode=0
Sep 17 14:08:00.040812 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:08:00.040778    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-749d46758f-gz2cr"
Sep 17 14:08:00.041260 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:08:00.040806    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-749d46758f-gz2cr" event={"ID":"47218685-e47f-4165-89e6-8dfc16504c60","Type":"ContainerDied","Data":"d37b3fc9c4ef789bbe66b6a4596cc182d2c209e4eecccc93465d94fc1cf1402d"}
Sep 17 14:08:00.041260 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:08:00.040849    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-749d46758f-gz2cr" event={"ID":"47218685-e47f-4165-89e6-8dfc16504c60","Type":"ContainerDied","Data":"e43dd36495a535866a8798e8a55e9ae3643986fbadd2947b157d5d08ce3cace8"}
Sep 17 14:08:00.041260 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:08:00.040867    2560 scope.go:117] "RemoveContainer" containerID="d37b3fc9c4ef789bbe66b6a4596cc182d2c209e4eecccc93465d94fc1cf1402d"
Sep 17 14:08:00.049679 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:08:00.049664    2560 scope.go:117] "RemoveContainer" containerID="d37b3fc9c4ef789bbe66b6a4596cc182d2c209e4eecccc93465d94fc1cf1402d"
Sep 17 14:08:00.049951 ip-10-0-129-158 kubenswrapper[2560]: E0917 14:08:00.049933    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d37b3fc9c4ef789bbe66b6a4596cc182d2c209e4eecccc93465d94fc1cf1402d\": container with ID starting with d37b3fc9c4ef789bbe66b6a4596cc182d2c209e4eecccc93465d94fc1cf1402d not found: ID does not exist" containerID="d37b3fc9c4ef789bbe66b6a4596cc182d2c209e4eecccc93465d94fc1cf1402d"
Sep 17 14:08:00.049998 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:08:00.049958    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d37b3fc9c4ef789bbe66b6a4596cc182d2c209e4eecccc93465d94fc1cf1402d"} err="failed to get container status \"d37b3fc9c4ef789bbe66b6a4596cc182d2c209e4eecccc93465d94fc1cf1402d\": rpc error: code = NotFound desc = could not find container \"d37b3fc9c4ef789bbe66b6a4596cc182d2c209e4eecccc93465d94fc1cf1402d\": container with ID starting with d37b3fc9c4ef789bbe66b6a4596cc182d2c209e4eecccc93465d94fc1cf1402d not found: ID does not exist"
Sep 17 14:08:00.064231 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:08:00.063881    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-749d46758f-gz2cr"]
Sep 17 14:08:00.064231 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:08:00.063925    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-749d46758f-gz2cr"]
Sep 17 14:08:01.569359 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:08:01.569314    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47218685-e47f-4165-89e6-8dfc16504c60" path="/var/lib/kubelet/pods/47218685-e47f-4165-89e6-8dfc16504c60/volumes"
Sep 17 14:08:01.892212 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:08:01.892115    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:08:02.634923 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:08:02.634892    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:08:02.735100 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:08:02.735064    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:08:05.219409 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:08:05.219358    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:08:05.325998 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:08:05.325965    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:08:29.502925 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:08:29.502881    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:08:36.971967 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:08:36.971923    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:08:37.094303 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:08:37.094268    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:08:38.445609 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:08:38.445574    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:08:40.155403 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:08:40.155369    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:08:40.251503 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:08:40.251474    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:08:49.756162 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:08:49.756123    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:08:49.857280 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:08:49.857241    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:00.719397 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:00.719367    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gdg6k"]
Sep 17 14:09:00.719884 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:00.719736    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94acd192-6343-431e-8711-f16dc6ef34d9" containerName="authorino"
Sep 17 14:09:00.719884 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:00.719748    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="94acd192-6343-431e-8711-f16dc6ef34d9" containerName="authorino"
Sep 17 14:09:00.719884 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:00.719761    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47218685-e47f-4165-89e6-8dfc16504c60" containerName="authorino"
Sep 17 14:09:00.719884 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:00.719766    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="47218685-e47f-4165-89e6-8dfc16504c60" containerName="authorino"
Sep 17 14:09:00.719884 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:00.719839    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="94acd192-6343-431e-8711-f16dc6ef34d9" containerName="authorino"
Sep 17 14:09:00.719884 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:00.719847    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="47218685-e47f-4165-89e6-8dfc16504c60" containerName="authorino"
Sep 17 14:09:00.722871 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:00.722852    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gdg6k"
Sep 17 14:09:00.725582 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:00.725566    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 17 14:09:00.726204 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:00.726187    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 17 14:09:00.726256 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:00.726204    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-98swh\""
Sep 17 14:09:00.726300 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:00.726204    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 17 14:09:00.734937 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:00.734914    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gdg6k"]
Sep 17 14:09:00.752019 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:00.751991    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:00.773843 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:00.773812    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1873d7b7-3c5b-46be-b6f1-757eb8c3b2e0-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gdg6k\" (UID: \"1873d7b7-3c5b-46be-b6f1-757eb8c3b2e0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gdg6k"
Sep 17 14:09:00.773958 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:00.773853    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1873d7b7-3c5b-46be-b6f1-757eb8c3b2e0-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gdg6k\" (UID: \"1873d7b7-3c5b-46be-b6f1-757eb8c3b2e0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gdg6k"
Sep 17 14:09:00.773958 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:00.773879    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd5nv\" (UniqueName: \"kubernetes.io/projected/1873d7b7-3c5b-46be-b6f1-757eb8c3b2e0-kube-api-access-kd5nv\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gdg6k\" (UID: \"1873d7b7-3c5b-46be-b6f1-757eb8c3b2e0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gdg6k"
Sep 17 14:09:00.773958 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:00.773948    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1873d7b7-3c5b-46be-b6f1-757eb8c3b2e0-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gdg6k\" (UID: \"1873d7b7-3c5b-46be-b6f1-757eb8c3b2e0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gdg6k"
Sep 17 14:09:00.774086 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:00.774036    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1873d7b7-3c5b-46be-b6f1-757eb8c3b2e0-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gdg6k\" (UID: \"1873d7b7-3c5b-46be-b6f1-757eb8c3b2e0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gdg6k"
Sep 17 14:09:00.774086 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:00.774064    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1873d7b7-3c5b-46be-b6f1-757eb8c3b2e0-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gdg6k\" (UID: \"1873d7b7-3c5b-46be-b6f1-757eb8c3b2e0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gdg6k"
Sep 17 14:09:00.875191 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:00.875169    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1873d7b7-3c5b-46be-b6f1-757eb8c3b2e0-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gdg6k\" (UID: \"1873d7b7-3c5b-46be-b6f1-757eb8c3b2e0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gdg6k"
Sep 17 14:09:00.875307 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:00.875200    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1873d7b7-3c5b-46be-b6f1-757eb8c3b2e0-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gdg6k\" (UID: \"1873d7b7-3c5b-46be-b6f1-757eb8c3b2e0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gdg6k"
Sep 17 14:09:00.875307 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:00.875231    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1873d7b7-3c5b-46be-b6f1-757eb8c3b2e0-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gdg6k\" (UID: \"1873d7b7-3c5b-46be-b6f1-757eb8c3b2e0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gdg6k"
Sep 17 14:09:00.875307 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:00.875249    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1873d7b7-3c5b-46be-b6f1-757eb8c3b2e0-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gdg6k\" (UID: \"1873d7b7-3c5b-46be-b6f1-757eb8c3b2e0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gdg6k"
Sep 17 14:09:00.875307 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:00.875275    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd5nv\" (UniqueName: \"kubernetes.io/projected/1873d7b7-3c5b-46be-b6f1-757eb8c3b2e0-kube-api-access-kd5nv\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gdg6k\" (UID: \"1873d7b7-3c5b-46be-b6f1-757eb8c3b2e0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gdg6k"
Sep 17 14:09:00.875307 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:00.875292    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1873d7b7-3c5b-46be-b6f1-757eb8c3b2e0-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gdg6k\" (UID: \"1873d7b7-3c5b-46be-b6f1-757eb8c3b2e0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gdg6k"
Sep 17 14:09:00.875627 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:00.875609    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1873d7b7-3c5b-46be-b6f1-757eb8c3b2e0-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gdg6k\" (UID: \"1873d7b7-3c5b-46be-b6f1-757eb8c3b2e0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gdg6k"
Sep 17 14:09:00.875685 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:00.875653    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1873d7b7-3c5b-46be-b6f1-757eb8c3b2e0-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gdg6k\" (UID: \"1873d7b7-3c5b-46be-b6f1-757eb8c3b2e0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gdg6k"
Sep 17 14:09:00.875685 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:00.875669    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1873d7b7-3c5b-46be-b6f1-757eb8c3b2e0-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gdg6k\" (UID: \"1873d7b7-3c5b-46be-b6f1-757eb8c3b2e0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gdg6k"
Sep 17 14:09:00.877394 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:00.877376    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1873d7b7-3c5b-46be-b6f1-757eb8c3b2e0-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gdg6k\" (UID: \"1873d7b7-3c5b-46be-b6f1-757eb8c3b2e0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gdg6k"
Sep 17 14:09:00.877508 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:00.877490    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1873d7b7-3c5b-46be-b6f1-757eb8c3b2e0-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gdg6k\" (UID: \"1873d7b7-3c5b-46be-b6f1-757eb8c3b2e0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gdg6k"
Sep 17 14:09:00.884913 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:00.884888    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd5nv\" (UniqueName: \"kubernetes.io/projected/1873d7b7-3c5b-46be-b6f1-757eb8c3b2e0-kube-api-access-kd5nv\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gdg6k\" (UID: \"1873d7b7-3c5b-46be-b6f1-757eb8c3b2e0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gdg6k"
Sep 17 14:09:01.033872 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:01.033801    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gdg6k"
Sep 17 14:09:01.311427 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:01.311401    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gdg6k"]
Sep 17 14:09:01.312704 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:09:01.312673    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1873d7b7_3c5b_46be_b6f1_757eb8c3b2e0.slice/crio-512baf4bd3c799d3836e7647e6c4ba77aa1c954f1a306c28b25008e5cf08e872 WatchSource:0}: Error finding container 512baf4bd3c799d3836e7647e6c4ba77aa1c954f1a306c28b25008e5cf08e872: Status 404 returned error can't find the container with id 512baf4bd3c799d3836e7647e6c4ba77aa1c954f1a306c28b25008e5cf08e872
Sep 17 14:09:01.314040 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:01.314022    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 14:09:01.367125 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:01.367095    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gdg6k" event={"ID":"1873d7b7-3c5b-46be-b6f1-757eb8c3b2e0","Type":"ContainerStarted","Data":"512baf4bd3c799d3836e7647e6c4ba77aa1c954f1a306c28b25008e5cf08e872"}
Sep 17 14:09:03.380186 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:03.380154    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gdg6k" event={"ID":"1873d7b7-3c5b-46be-b6f1-757eb8c3b2e0","Type":"ContainerStarted","Data":"c14dc13c7331a94e8d4c974c68ec11cf054f3b250d682f6c9de86b5dd69fcff4"}
Sep 17 14:09:03.402664 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:03.402614    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gdg6k" podStartSLOduration=2.351870228 podStartE2EDuration="3.402598265s" totalImagesPullingTime="1.050728037s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:09:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:09:03.40142855 +0000 UTC m=+812.343572190" watchObservedRunningTime="2026-09-17 14:09:03.402598265 +0000 UTC m=+812.344741910"
Sep 17 14:09:07.715630 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:07.715593    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-msqdt"]
Sep 17 14:09:07.719202 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:07.719186    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-msqdt"
Sep 17 14:09:07.721199 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:07.721179    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 17 14:09:07.727993 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:07.727971    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-msqdt"]
Sep 17 14:09:07.817642 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:07.817612    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:07.830668 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:07.830639    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68fc6c1d-a850-4223-be13-27c9d4c47625-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-msqdt\" (UID: \"68fc6c1d-a850-4223-be13-27c9d4c47625\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-msqdt"
Sep 17 14:09:07.830668 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:07.830670    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68fc6c1d-a850-4223-be13-27c9d4c47625-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-msqdt\" (UID: \"68fc6c1d-a850-4223-be13-27c9d4c47625\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-msqdt"
Sep 17 14:09:07.830836 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:07.830686    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8nrd\" (UniqueName: \"kubernetes.io/projected/68fc6c1d-a850-4223-be13-27c9d4c47625-kube-api-access-b8nrd\") pod \"e2e-distinct-simulated-kserve-59975d69f9-msqdt\" (UID: \"68fc6c1d-a850-4223-be13-27c9d4c47625\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-msqdt"
Sep 17 14:09:07.830836 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:07.830807    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68fc6c1d-a850-4223-be13-27c9d4c47625-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-msqdt\" (UID: \"68fc6c1d-a850-4223-be13-27c9d4c47625\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-msqdt"
Sep 17 14:09:07.830919 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:07.830839    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68fc6c1d-a850-4223-be13-27c9d4c47625-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-msqdt\" (UID: \"68fc6c1d-a850-4223-be13-27c9d4c47625\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-msqdt"
Sep 17 14:09:07.830957 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:07.830922    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68fc6c1d-a850-4223-be13-27c9d4c47625-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-msqdt\" (UID: \"68fc6c1d-a850-4223-be13-27c9d4c47625\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-msqdt"
Sep 17 14:09:07.931635 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:07.931608    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68fc6c1d-a850-4223-be13-27c9d4c47625-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-msqdt\" (UID: \"68fc6c1d-a850-4223-be13-27c9d4c47625\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-msqdt"
Sep 17 14:09:07.931792 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:07.931646    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68fc6c1d-a850-4223-be13-27c9d4c47625-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-msqdt\" (UID: \"68fc6c1d-a850-4223-be13-27c9d4c47625\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-msqdt"
Sep 17 14:09:07.931792 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:07.931684    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68fc6c1d-a850-4223-be13-27c9d4c47625-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-msqdt\" (UID: \"68fc6c1d-a850-4223-be13-27c9d4c47625\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-msqdt"
Sep 17 14:09:07.931792 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:07.931703    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68fc6c1d-a850-4223-be13-27c9d4c47625-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-msqdt\" (UID: \"68fc6c1d-a850-4223-be13-27c9d4c47625\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-msqdt"
Sep 17 14:09:07.931792 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:07.931725    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68fc6c1d-a850-4223-be13-27c9d4c47625-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-msqdt\" (UID: \"68fc6c1d-a850-4223-be13-27c9d4c47625\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-msqdt"
Sep 17 14:09:07.931792 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:07.931748    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8nrd\" (UniqueName: \"kubernetes.io/projected/68fc6c1d-a850-4223-be13-27c9d4c47625-kube-api-access-b8nrd\") pod \"e2e-distinct-simulated-kserve-59975d69f9-msqdt\" (UID: \"68fc6c1d-a850-4223-be13-27c9d4c47625\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-msqdt"
Sep 17 14:09:07.932055 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:07.931995    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68fc6c1d-a850-4223-be13-27c9d4c47625-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-msqdt\" (UID: \"68fc6c1d-a850-4223-be13-27c9d4c47625\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-msqdt"
Sep 17 14:09:07.932122 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:07.932101    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68fc6c1d-a850-4223-be13-27c9d4c47625-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-msqdt\" (UID: \"68fc6c1d-a850-4223-be13-27c9d4c47625\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-msqdt"
Sep 17 14:09:07.932162 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:07.932132    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68fc6c1d-a850-4223-be13-27c9d4c47625-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-msqdt\" (UID: \"68fc6c1d-a850-4223-be13-27c9d4c47625\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-msqdt"
Sep 17 14:09:07.933878 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:07.933861    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68fc6c1d-a850-4223-be13-27c9d4c47625-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-msqdt\" (UID: \"68fc6c1d-a850-4223-be13-27c9d4c47625\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-msqdt"
Sep 17 14:09:07.934045 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:07.934028    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68fc6c1d-a850-4223-be13-27c9d4c47625-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-msqdt\" (UID: \"68fc6c1d-a850-4223-be13-27c9d4c47625\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-msqdt"
Sep 17 14:09:07.939774 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:07.939749    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8nrd\" (UniqueName: \"kubernetes.io/projected/68fc6c1d-a850-4223-be13-27c9d4c47625-kube-api-access-b8nrd\") pod \"e2e-distinct-simulated-kserve-59975d69f9-msqdt\" (UID: \"68fc6c1d-a850-4223-be13-27c9d4c47625\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-msqdt"
Sep 17 14:09:08.030227 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:08.030164    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-msqdt"
Sep 17 14:09:08.303452 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:08.303427    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-msqdt"]
Sep 17 14:09:08.305109 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:09:08.305086    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68fc6c1d_a850_4223_be13_27c9d4c47625.slice/crio-084a32fa881efb3a4fdfc1f2f9e5fb28b0ea258987b0d822418d39a1922cea0a WatchSource:0}: Error finding container 084a32fa881efb3a4fdfc1f2f9e5fb28b0ea258987b0d822418d39a1922cea0a: Status 404 returned error can't find the container with id 084a32fa881efb3a4fdfc1f2f9e5fb28b0ea258987b0d822418d39a1922cea0a
Sep 17 14:09:08.410218 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:08.410187    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-msqdt" event={"ID":"68fc6c1d-a850-4223-be13-27c9d4c47625","Type":"ContainerStarted","Data":"084a32fa881efb3a4fdfc1f2f9e5fb28b0ea258987b0d822418d39a1922cea0a"}
Sep 17 14:09:09.417196 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:09.417162    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-msqdt" event={"ID":"68fc6c1d-a850-4223-be13-27c9d4c47625","Type":"ContainerStarted","Data":"dceca5ab266f0a2847faabac2fb98c64ab1b057006e5ebbadeaf6e3ba88a65b9"}
Sep 17 14:09:09.436931 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:09.436887    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-msqdt" podStartSLOduration=2.259963699 podStartE2EDuration="2.436873195s" totalImagesPullingTime="176.909496ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:09:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:09:09.43466054 +0000 UTC m=+818.376804186" watchObservedRunningTime="2026-09-17 14:09:09.436873195 +0000 UTC m=+818.379016841"
Sep 17 14:09:11.034698 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:11.034663    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gdg6k"
Sep 17 14:09:11.035106 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:11.034718    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gdg6k"
Sep 17 14:09:11.046900 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:11.046878    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gdg6k"
Sep 17 14:09:11.438374 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:11.438346    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gdg6k"
Sep 17 14:09:14.960405 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:14.960369    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:18.030924 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:18.030886    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-msqdt"
Sep 17 14:09:18.031305 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:18.030949    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-msqdt"
Sep 17 14:09:18.043444 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:18.043421    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-msqdt"
Sep 17 14:09:18.476369 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:18.476340    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-msqdt"
Sep 17 14:09:29.349506 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:29.349446    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:39.237615 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:39.237561    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:43.922148 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:43.922117    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:51.918711 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:51.918678    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-fdz9h"]
Sep 17 14:09:51.922519 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:51.922502    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fdz9h"
Sep 17 14:09:51.925037 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:51.925018    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 17 14:09:51.934347 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:51.934326    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-fdz9h"]
Sep 17 14:09:51.996416 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:51.996387    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bf20d2c-b82c-4536-8f83-f64992c0bb32-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fdz9h\" (UID: \"9bf20d2c-b82c-4536-8f83-f64992c0bb32\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fdz9h"
Sep 17 14:09:51.996545 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:51.996418    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5km4\" (UniqueName: \"kubernetes.io/projected/9bf20d2c-b82c-4536-8f83-f64992c0bb32-kube-api-access-l5km4\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fdz9h\" (UID: \"9bf20d2c-b82c-4536-8f83-f64992c0bb32\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fdz9h"
Sep 17 14:09:51.996545 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:51.996539    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bf20d2c-b82c-4536-8f83-f64992c0bb32-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fdz9h\" (UID: \"9bf20d2c-b82c-4536-8f83-f64992c0bb32\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fdz9h"
Sep 17 14:09:51.996645 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:51.996572    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bf20d2c-b82c-4536-8f83-f64992c0bb32-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fdz9h\" (UID: \"9bf20d2c-b82c-4536-8f83-f64992c0bb32\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fdz9h"
Sep 17 14:09:51.996645 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:51.996607    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bf20d2c-b82c-4536-8f83-f64992c0bb32-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fdz9h\" (UID: \"9bf20d2c-b82c-4536-8f83-f64992c0bb32\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fdz9h"
Sep 17 14:09:51.996645 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:51.996636    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bf20d2c-b82c-4536-8f83-f64992c0bb32-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fdz9h\" (UID: \"9bf20d2c-b82c-4536-8f83-f64992c0bb32\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fdz9h"
Sep 17 14:09:52.026302 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:52.026273    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:52.097808 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:52.097781    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bf20d2c-b82c-4536-8f83-f64992c0bb32-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fdz9h\" (UID: \"9bf20d2c-b82c-4536-8f83-f64992c0bb32\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fdz9h"
Sep 17 14:09:52.097922 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:52.097816    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5km4\" (UniqueName: \"kubernetes.io/projected/9bf20d2c-b82c-4536-8f83-f64992c0bb32-kube-api-access-l5km4\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fdz9h\" (UID: \"9bf20d2c-b82c-4536-8f83-f64992c0bb32\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fdz9h"
Sep 17 14:09:52.097922 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:52.097867    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bf20d2c-b82c-4536-8f83-f64992c0bb32-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fdz9h\" (UID: \"9bf20d2c-b82c-4536-8f83-f64992c0bb32\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fdz9h"
Sep 17 14:09:52.097922 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:52.097905    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bf20d2c-b82c-4536-8f83-f64992c0bb32-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fdz9h\" (UID: \"9bf20d2c-b82c-4536-8f83-f64992c0bb32\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fdz9h"
Sep 17 14:09:52.098082 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:52.097937    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bf20d2c-b82c-4536-8f83-f64992c0bb32-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fdz9h\" (UID: \"9bf20d2c-b82c-4536-8f83-f64992c0bb32\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fdz9h"
Sep 17 14:09:52.098082 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:52.097974    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bf20d2c-b82c-4536-8f83-f64992c0bb32-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fdz9h\" (UID: \"9bf20d2c-b82c-4536-8f83-f64992c0bb32\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fdz9h"
Sep 17 14:09:52.098186 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:52.098159    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bf20d2c-b82c-4536-8f83-f64992c0bb32-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fdz9h\" (UID: \"9bf20d2c-b82c-4536-8f83-f64992c0bb32\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fdz9h"
Sep 17 14:09:52.098242 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:52.098223    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bf20d2c-b82c-4536-8f83-f64992c0bb32-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fdz9h\" (UID: \"9bf20d2c-b82c-4536-8f83-f64992c0bb32\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fdz9h"
Sep 17 14:09:52.098325 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:52.098302    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bf20d2c-b82c-4536-8f83-f64992c0bb32-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fdz9h\" (UID: \"9bf20d2c-b82c-4536-8f83-f64992c0bb32\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fdz9h"
Sep 17 14:09:52.100017 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:52.099999    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bf20d2c-b82c-4536-8f83-f64992c0bb32-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fdz9h\" (UID: \"9bf20d2c-b82c-4536-8f83-f64992c0bb32\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fdz9h"
Sep 17 14:09:52.100340 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:52.100323    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bf20d2c-b82c-4536-8f83-f64992c0bb32-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fdz9h\" (UID: \"9bf20d2c-b82c-4536-8f83-f64992c0bb32\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fdz9h"
Sep 17 14:09:52.106707 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:52.106687    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5km4\" (UniqueName: \"kubernetes.io/projected/9bf20d2c-b82c-4536-8f83-f64992c0bb32-kube-api-access-l5km4\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fdz9h\" (UID: \"9bf20d2c-b82c-4536-8f83-f64992c0bb32\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fdz9h"
Sep 17 14:09:52.233199 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:52.233132    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fdz9h"
Sep 17 14:09:52.505983 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:52.505955    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-fdz9h"]
Sep 17 14:09:52.508783 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:09:52.508757    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bf20d2c_b82c_4536_8f83_f64992c0bb32.slice/crio-1cf7eefb0eadbce860be0c16e8c2a3af3ab13d7a7e2a88ea396ede2f12edca44 WatchSource:0}: Error finding container 1cf7eefb0eadbce860be0c16e8c2a3af3ab13d7a7e2a88ea396ede2f12edca44: Status 404 returned error can't find the container with id 1cf7eefb0eadbce860be0c16e8c2a3af3ab13d7a7e2a88ea396ede2f12edca44
Sep 17 14:09:52.648338 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:52.648306    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fdz9h" event={"ID":"9bf20d2c-b82c-4536-8f83-f64992c0bb32","Type":"ContainerStarted","Data":"1cf7eefb0eadbce860be0c16e8c2a3af3ab13d7a7e2a88ea396ede2f12edca44"}
Sep 17 14:09:53.655106 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:53.655073    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fdz9h" event={"ID":"9bf20d2c-b82c-4536-8f83-f64992c0bb32","Type":"ContainerStarted","Data":"cb1304f0fc6d9ae1cb4c37f093bd0b164dd0d3cd098a05e3e2100217bf0267c2"}
Sep 17 14:09:53.677257 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:53.677211    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fdz9h" podStartSLOduration=2.480838166 podStartE2EDuration="2.677195983s" totalImagesPullingTime="196.357817ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:09:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:09:53.674682243 +0000 UTC m=+862.616825891" watchObservedRunningTime="2026-09-17 14:09:53.677195983 +0000 UTC m=+862.619339629"
Sep 17 14:09:53.806301 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:09:53.806267    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:10:02.233807 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:10:02.233761    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fdz9h"
Sep 17 14:10:02.234241 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:10:02.233843    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fdz9h"
Sep 17 14:10:02.245750 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:10:02.245733    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fdz9h"
Sep 17 14:10:02.715932 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:10:02.715905    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fdz9h"
Sep 17 14:10:13.640588 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:10:13.640552    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:10:59.511334 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:10:59.511300    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:10.435845 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:10.435803    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:10.538332 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:10.538288    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:11.494754 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:11.494717    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:11.595688 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:11.595653    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:27.315297 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:27.315258    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:27.412502 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:27.412473    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:27.704062 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:27.703973    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4"]
Sep 17 14:11:27.707676 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:27.707653    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4"
Sep 17 14:11:27.710682 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:27.710659    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-96cfc6-openshift-default-dockercfg-ls25d\""
Sep 17 14:11:27.726961 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:27.726921    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4"]
Sep 17 14:11:27.783895 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:27.783860    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s"]
Sep 17 14:11:27.787439 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:27.787421    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s"
Sep 17 14:11:27.790632 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:27.790614    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-d9b17d-openshift-default-dockercfg-xl84g\""
Sep 17 14:11:27.798779 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:27.798754    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08fb3245-86c7-4cab-86d9-3b5177ca5101-workload-certs\") pod \"e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4\" (UID: \"08fb3245-86c7-4cab-86d9-3b5177ca5101\") " pod="openshift-ingress/e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4"
Sep 17 14:11:27.798875 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:27.798792    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08fb3245-86c7-4cab-86d9-3b5177ca5101-istio-envoy\") pod \"e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4\" (UID: \"08fb3245-86c7-4cab-86d9-3b5177ca5101\") " pod="openshift-ingress/e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4"
Sep 17 14:11:27.798875 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:27.798825    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08fb3245-86c7-4cab-86d9-3b5177ca5101-istio-data\") pod \"e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4\" (UID: \"08fb3245-86c7-4cab-86d9-3b5177ca5101\") " pod="openshift-ingress/e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4"
Sep 17 14:11:27.799000 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:27.798906    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08fb3245-86c7-4cab-86d9-3b5177ca5101-istiod-ca-cert\") pod \"e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4\" (UID: \"08fb3245-86c7-4cab-86d9-3b5177ca5101\") " pod="openshift-ingress/e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4"
Sep 17 14:11:27.799000 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:27.798967    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08fb3245-86c7-4cab-86d9-3b5177ca5101-istio-podinfo\") pod \"e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4\" (UID: \"08fb3245-86c7-4cab-86d9-3b5177ca5101\") " pod="openshift-ingress/e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4"
Sep 17 14:11:27.799102 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:27.798997    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmkxc\" (UniqueName: \"kubernetes.io/projected/08fb3245-86c7-4cab-86d9-3b5177ca5101-kube-api-access-mmkxc\") pod \"e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4\" (UID: \"08fb3245-86c7-4cab-86d9-3b5177ca5101\") " pod="openshift-ingress/e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4"
Sep 17 14:11:27.799102 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:27.799026    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08fb3245-86c7-4cab-86d9-3b5177ca5101-credential-socket\") pod \"e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4\" (UID: \"08fb3245-86c7-4cab-86d9-3b5177ca5101\") " pod="openshift-ingress/e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4"
Sep 17 14:11:27.799102 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:27.799091    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08fb3245-86c7-4cab-86d9-3b5177ca5101-istio-token\") pod \"e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4\" (UID: \"08fb3245-86c7-4cab-86d9-3b5177ca5101\") " pod="openshift-ingress/e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4"
Sep 17 14:11:27.799290 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:27.799119    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08fb3245-86c7-4cab-86d9-3b5177ca5101-workload-socket\") pod \"e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4\" (UID: \"08fb3245-86c7-4cab-86d9-3b5177ca5101\") " pod="openshift-ingress/e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4"
Sep 17 14:11:27.799634 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:27.799616    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s"]
Sep 17 14:11:27.900259 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:27.900219    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47668435-ed43-4f78-a71b-ceb972a66e25-workload-certs\") pod \"e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s\" (UID: \"47668435-ed43-4f78-a71b-ceb972a66e25\") " pod="openshift-ingress/e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s"
Sep 17 14:11:27.900576 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:27.900542    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08fb3245-86c7-4cab-86d9-3b5177ca5101-istio-token\") pod \"e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4\" (UID: \"08fb3245-86c7-4cab-86d9-3b5177ca5101\") " pod="openshift-ingress/e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4"
Sep 17 14:11:27.900745 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:27.900587    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47668435-ed43-4f78-a71b-ceb972a66e25-credential-socket\") pod \"e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s\" (UID: \"47668435-ed43-4f78-a71b-ceb972a66e25\") " pod="openshift-ingress/e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s"
Sep 17 14:11:27.900745 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:27.900616    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47668435-ed43-4f78-a71b-ceb972a66e25-istio-podinfo\") pod \"e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s\" (UID: \"47668435-ed43-4f78-a71b-ceb972a66e25\") " pod="openshift-ingress/e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s"
Sep 17 14:11:27.900745 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:27.900640    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08fb3245-86c7-4cab-86d9-3b5177ca5101-workload-socket\") pod \"e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4\" (UID: \"08fb3245-86c7-4cab-86d9-3b5177ca5101\") " pod="openshift-ingress/e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4"
Sep 17 14:11:27.900940 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:27.900755    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08fb3245-86c7-4cab-86d9-3b5177ca5101-workload-certs\") pod \"e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4\" (UID: \"08fb3245-86c7-4cab-86d9-3b5177ca5101\") " pod="openshift-ingress/e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4"
Sep 17 14:11:27.900940 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:27.900797    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08fb3245-86c7-4cab-86d9-3b5177ca5101-istio-envoy\") pod \"e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4\" (UID: \"08fb3245-86c7-4cab-86d9-3b5177ca5101\") " pod="openshift-ingress/e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4"
Sep 17 14:11:27.900940 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:27.900834    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08fb3245-86c7-4cab-86d9-3b5177ca5101-istio-data\") pod \"e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4\" (UID: \"08fb3245-86c7-4cab-86d9-3b5177ca5101\") " pod="openshift-ingress/e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4"
Sep 17 14:11:27.900940 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:27.900858    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47668435-ed43-4f78-a71b-ceb972a66e25-istio-data\") pod \"e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s\" (UID: \"47668435-ed43-4f78-a71b-ceb972a66e25\") " pod="openshift-ingress/e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s"
Sep 17 14:11:27.900940 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:27.900894    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47668435-ed43-4f78-a71b-ceb972a66e25-istio-token\") pod \"e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s\" (UID: \"47668435-ed43-4f78-a71b-ceb972a66e25\") " pod="openshift-ingress/e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s"
Sep 17 14:11:27.900940 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:27.900927    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47668435-ed43-4f78-a71b-ceb972a66e25-istio-envoy\") pod \"e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s\" (UID: \"47668435-ed43-4f78-a71b-ceb972a66e25\") " pod="openshift-ingress/e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s"
Sep 17 14:11:27.901324 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:27.900971    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47668435-ed43-4f78-a71b-ceb972a66e25-workload-socket\") pod \"e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s\" (UID: \"47668435-ed43-4f78-a71b-ceb972a66e25\") " pod="openshift-ingress/e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s"
Sep 17 14:11:27.901324 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:27.901018    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08fb3245-86c7-4cab-86d9-3b5177ca5101-istiod-ca-cert\") pod \"e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4\" (UID: \"08fb3245-86c7-4cab-86d9-3b5177ca5101\") " pod="openshift-ingress/e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4"
Sep 17 14:11:27.901324 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:27.901044    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47668435-ed43-4f78-a71b-ceb972a66e25-istiod-ca-cert\") pod \"e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s\" (UID: \"47668435-ed43-4f78-a71b-ceb972a66e25\") " pod="openshift-ingress/e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s"
Sep 17 14:11:27.901324 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:27.901105    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lztr9\" (UniqueName: \"kubernetes.io/projected/47668435-ed43-4f78-a71b-ceb972a66e25-kube-api-access-lztr9\") pod \"e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s\" (UID: \"47668435-ed43-4f78-a71b-ceb972a66e25\") " pod="openshift-ingress/e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s"
Sep 17 14:11:27.901324 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:27.901143    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08fb3245-86c7-4cab-86d9-3b5177ca5101-istio-podinfo\") pod \"e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4\" (UID: \"08fb3245-86c7-4cab-86d9-3b5177ca5101\") " pod="openshift-ingress/e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4"
Sep 17 14:11:27.901324 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:27.901180    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmkxc\" (UniqueName: \"kubernetes.io/projected/08fb3245-86c7-4cab-86d9-3b5177ca5101-kube-api-access-mmkxc\") pod \"e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4\" (UID: \"08fb3245-86c7-4cab-86d9-3b5177ca5101\") " pod="openshift-ingress/e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4"
Sep 17 14:11:27.901324 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:27.901217    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08fb3245-86c7-4cab-86d9-3b5177ca5101-credential-socket\") pod \"e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4\" (UID: \"08fb3245-86c7-4cab-86d9-3b5177ca5101\") " pod="openshift-ingress/e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4"
Sep 17 14:11:27.901718 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:27.901441    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08fb3245-86c7-4cab-86d9-3b5177ca5101-istio-data\") pod \"e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4\" (UID: \"08fb3245-86c7-4cab-86d9-3b5177ca5101\") " pod="openshift-ingress/e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4"
Sep 17 14:11:27.902033 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:27.902013    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08fb3245-86c7-4cab-86d9-3b5177ca5101-credential-socket\") pod \"e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4\" (UID: \"08fb3245-86c7-4cab-86d9-3b5177ca5101\") " pod="openshift-ingress/e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4"
Sep 17 14:11:27.902193 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:27.902023    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08fb3245-86c7-4cab-86d9-3b5177ca5101-workload-socket\") pod \"e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4\" (UID: \"08fb3245-86c7-4cab-86d9-3b5177ca5101\") " pod="openshift-ingress/e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4"
Sep 17 14:11:27.902289 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:27.902242    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08fb3245-86c7-4cab-86d9-3b5177ca5101-workload-certs\") pod \"e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4\" (UID: \"08fb3245-86c7-4cab-86d9-3b5177ca5101\") " pod="openshift-ingress/e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4"
Sep 17 14:11:27.903067 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:27.903043    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08fb3245-86c7-4cab-86d9-3b5177ca5101-istiod-ca-cert\") pod \"e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4\" (UID: \"08fb3245-86c7-4cab-86d9-3b5177ca5101\") " pod="openshift-ingress/e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4"
Sep 17 14:11:27.904175 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:27.904137    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08fb3245-86c7-4cab-86d9-3b5177ca5101-istio-envoy\") pod \"e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4\" (UID: \"08fb3245-86c7-4cab-86d9-3b5177ca5101\") " pod="openshift-ingress/e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4"
Sep 17 14:11:27.904905 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:27.904885    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08fb3245-86c7-4cab-86d9-3b5177ca5101-istio-podinfo\") pod \"e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4\" (UID: \"08fb3245-86c7-4cab-86d9-3b5177ca5101\") " pod="openshift-ingress/e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4"
Sep 17 14:11:27.908840 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:27.908817    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmkxc\" (UniqueName: \"kubernetes.io/projected/08fb3245-86c7-4cab-86d9-3b5177ca5101-kube-api-access-mmkxc\") pod \"e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4\" (UID: \"08fb3245-86c7-4cab-86d9-3b5177ca5101\") " pod="openshift-ingress/e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4"
Sep 17 14:11:27.908977 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:27.908963    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08fb3245-86c7-4cab-86d9-3b5177ca5101-istio-token\") pod \"e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4\" (UID: \"08fb3245-86c7-4cab-86d9-3b5177ca5101\") " pod="openshift-ingress/e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4"
Sep 17 14:11:28.002242 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:28.002160    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47668435-ed43-4f78-a71b-ceb972a66e25-credential-socket\") pod \"e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s\" (UID: \"47668435-ed43-4f78-a71b-ceb972a66e25\") " pod="openshift-ingress/e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s"
Sep 17 14:11:28.002242 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:28.002192    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47668435-ed43-4f78-a71b-ceb972a66e25-istio-podinfo\") pod \"e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s\" (UID: \"47668435-ed43-4f78-a71b-ceb972a66e25\") " pod="openshift-ingress/e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s"
Sep 17 14:11:28.002242 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:28.002229    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47668435-ed43-4f78-a71b-ceb972a66e25-istio-data\") pod \"e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s\" (UID: \"47668435-ed43-4f78-a71b-ceb972a66e25\") " pod="openshift-ingress/e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s"
Sep 17 14:11:28.002574 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:28.002251    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47668435-ed43-4f78-a71b-ceb972a66e25-istio-token\") pod \"e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s\" (UID: \"47668435-ed43-4f78-a71b-ceb972a66e25\") " pod="openshift-ingress/e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s"
Sep 17 14:11:28.002574 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:28.002271    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47668435-ed43-4f78-a71b-ceb972a66e25-istio-envoy\") pod \"e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s\" (UID: \"47668435-ed43-4f78-a71b-ceb972a66e25\") " pod="openshift-ingress/e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s"
Sep 17 14:11:28.002574 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:28.002302    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47668435-ed43-4f78-a71b-ceb972a66e25-workload-socket\") pod \"e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s\" (UID: \"47668435-ed43-4f78-a71b-ceb972a66e25\") " pod="openshift-ingress/e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s"
Sep 17 14:11:28.002574 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:28.002326    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47668435-ed43-4f78-a71b-ceb972a66e25-istiod-ca-cert\") pod \"e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s\" (UID: \"47668435-ed43-4f78-a71b-ceb972a66e25\") " pod="openshift-ingress/e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s"
Sep 17 14:11:28.002574 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:28.002368    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lztr9\" (UniqueName: \"kubernetes.io/projected/47668435-ed43-4f78-a71b-ceb972a66e25-kube-api-access-lztr9\") pod \"e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s\" (UID: \"47668435-ed43-4f78-a71b-ceb972a66e25\") " pod="openshift-ingress/e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s"
Sep 17 14:11:28.002574 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:28.002426    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47668435-ed43-4f78-a71b-ceb972a66e25-workload-certs\") pod \"e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s\" (UID: \"47668435-ed43-4f78-a71b-ceb972a66e25\") " pod="openshift-ingress/e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s"
Sep 17 14:11:28.002873 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:28.002645    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47668435-ed43-4f78-a71b-ceb972a66e25-credential-socket\") pod \"e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s\" (UID: \"47668435-ed43-4f78-a71b-ceb972a66e25\") " pod="openshift-ingress/e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s"
Sep 17 14:11:28.002873 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:28.002764    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47668435-ed43-4f78-a71b-ceb972a66e25-workload-socket\") pod \"e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s\" (UID: \"47668435-ed43-4f78-a71b-ceb972a66e25\") " pod="openshift-ingress/e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s"
Sep 17 14:11:28.002873 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:28.002839    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47668435-ed43-4f78-a71b-ceb972a66e25-workload-certs\") pod \"e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s\" (UID: \"47668435-ed43-4f78-a71b-ceb972a66e25\") " pod="openshift-ingress/e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s"
Sep 17 14:11:28.003034 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:28.002873    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47668435-ed43-4f78-a71b-ceb972a66e25-istio-data\") pod \"e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s\" (UID: \"47668435-ed43-4f78-a71b-ceb972a66e25\") " pod="openshift-ingress/e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s"
Sep 17 14:11:28.003034 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:28.003001    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47668435-ed43-4f78-a71b-ceb972a66e25-istiod-ca-cert\") pod \"e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s\" (UID: \"47668435-ed43-4f78-a71b-ceb972a66e25\") " pod="openshift-ingress/e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s"
Sep 17 14:11:28.004606 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:28.004580    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47668435-ed43-4f78-a71b-ceb972a66e25-istio-envoy\") pod \"e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s\" (UID: \"47668435-ed43-4f78-a71b-ceb972a66e25\") " pod="openshift-ingress/e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s"
Sep 17 14:11:28.004965 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:28.004943    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47668435-ed43-4f78-a71b-ceb972a66e25-istio-podinfo\") pod \"e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s\" (UID: \"47668435-ed43-4f78-a71b-ceb972a66e25\") " pod="openshift-ingress/e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s"
Sep 17 14:11:28.010108 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:28.010088    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47668435-ed43-4f78-a71b-ceb972a66e25-istio-token\") pod \"e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s\" (UID: \"47668435-ed43-4f78-a71b-ceb972a66e25\") " pod="openshift-ingress/e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s"
Sep 17 14:11:28.010290 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:28.010272    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lztr9\" (UniqueName: \"kubernetes.io/projected/47668435-ed43-4f78-a71b-ceb972a66e25-kube-api-access-lztr9\") pod \"e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s\" (UID: \"47668435-ed43-4f78-a71b-ceb972a66e25\") " pod="openshift-ingress/e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s"
Sep 17 14:11:28.020190 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:28.020152    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4"
Sep 17 14:11:28.098692 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:28.098663    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s"
Sep 17 14:11:28.300700 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:28.300670    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4"]
Sep 17 14:11:28.302122 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:11:28.302099    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08fb3245_86c7_4cab_86d9_3b5177ca5101.slice/crio-15fca760cfbc68055cfe475d208d6e7655d28191e00b381a59b69cc9e8829a4f WatchSource:0}: Error finding container 15fca760cfbc68055cfe475d208d6e7655d28191e00b381a59b69cc9e8829a4f: Status 404 returned error can't find the container with id 15fca760cfbc68055cfe475d208d6e7655d28191e00b381a59b69cc9e8829a4f
Sep 17 14:11:28.304241 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:28.304209    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:11:28.304337 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:28.304286    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:11:28.304404 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:28.304333    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:11:28.380172 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:28.380139    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s"]
Sep 17 14:11:28.382192 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:11:28.382159    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47668435_ed43_4f78_a71b_ceb972a66e25.slice/crio-ab01864c94ef2561b403eac44b245743b4139dfbf354ced6ac634286b493558c WatchSource:0}: Error finding container ab01864c94ef2561b403eac44b245743b4139dfbf354ced6ac634286b493558c: Status 404 returned error can't find the container with id ab01864c94ef2561b403eac44b245743b4139dfbf354ced6ac634286b493558c
Sep 17 14:11:28.384620 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:28.384587    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:11:28.384712 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:28.384651    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:11:28.384712 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:28.384706    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:11:28.935122 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:28.935089    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:29.029639 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:29.029599    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:29.189414 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:29.189323    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4" event={"ID":"08fb3245-86c7-4cab-86d9-3b5177ca5101","Type":"ContainerStarted","Data":"3a6f61ffc5af81c362c8f2df3e6e6509b26ef1fe03f530db443d6b30a487fc7f"}
Sep 17 14:11:29.189414 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:29.189380    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4" event={"ID":"08fb3245-86c7-4cab-86d9-3b5177ca5101","Type":"ContainerStarted","Data":"15fca760cfbc68055cfe475d208d6e7655d28191e00b381a59b69cc9e8829a4f"}
Sep 17 14:11:29.190928 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:29.190902    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s" event={"ID":"47668435-ed43-4f78-a71b-ceb972a66e25","Type":"ContainerStarted","Data":"cd85126c847fb7d5a7f45f2cc20ca0a4936ad72022acd9bca6c3598ed7c0d204"}
Sep 17 14:11:29.191057 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:29.190940    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s" event={"ID":"47668435-ed43-4f78-a71b-ceb972a66e25","Type":"ContainerStarted","Data":"ab01864c94ef2561b403eac44b245743b4139dfbf354ced6ac634286b493558c"}
Sep 17 14:11:29.214187 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:29.214144    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4" podStartSLOduration=2.21412944 podStartE2EDuration="2.21412944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:11:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:11:29.210660583 +0000 UTC m=+958.152804228" watchObservedRunningTime="2026-09-17 14:11:29.21412944 +0000 UTC m=+958.156273086"
Sep 17 14:11:29.237664 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:29.237613    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s" podStartSLOduration=2.237595004 podStartE2EDuration="2.237595004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:11:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:11:29.234325324 +0000 UTC m=+958.176468972" watchObservedRunningTime="2026-09-17 14:11:29.237595004 +0000 UTC m=+958.179738650"
Sep 17 14:11:29.245113 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:29.245086    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:30.021291 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:30.021252    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4"
Sep 17 14:11:30.026388 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:30.026365    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4"
Sep 17 14:11:30.099568 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:30.099534    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s"
Sep 17 14:11:30.104215 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:30.104192    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s"
Sep 17 14:11:30.196805 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:30.196766    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4"
Sep 17 14:11:30.197004 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:30.196833    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s"
Sep 17 14:11:30.197765 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:30.197743    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4"
Sep 17 14:11:30.197890 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:30.197789    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s"
Sep 17 14:11:30.887480 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:30.887430    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:30.984100 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:30.984070    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:31.258162 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:31.258076    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:32.804143 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:32.804110    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:32.922310 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:32.922277    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:33.340427 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:33.340394    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:33.440868 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:33.440829    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:33.513924 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:33.513895    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:33.836550 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:33.836512    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:33.938090 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:33.938056    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:34.035714 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:34.035680    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:34.344347 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:34.344313    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:34.450024 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:34.449989    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:34.535027 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:34.534998    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-d9b17d-67659db75d-fl9f9"]
Sep 17 14:11:34.539727 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:34.539704    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d9b17d-67659db75d-fl9f9"
Sep 17 14:11:34.542395 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:34.542371    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-d9b17d-dockercfg-52wjq\""
Sep 17 14:11:34.542395 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:34.542390    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-d9b17d\""
Sep 17 14:11:34.548268 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:34.548241    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-d9b17d-67659db75d-fl9f9"]
Sep 17 14:11:34.670283 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:34.670190    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87759bfd-b7d8-45e0-b5f0-8d5c081abaf3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-d9b17d-67659db75d-fl9f9\" (UID: \"87759bfd-b7d8-45e0-b5f0-8d5c081abaf3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d9b17d-67659db75d-fl9f9"
Sep 17 14:11:34.670439 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:34.670319    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rfxq\" (UniqueName: \"kubernetes.io/projected/87759bfd-b7d8-45e0-b5f0-8d5c081abaf3-kube-api-access-2rfxq\") pod \"payload-pre-processing-e2e-worker-w0-d9b17d-67659db75d-fl9f9\" (UID: \"87759bfd-b7d8-45e0-b5f0-8d5c081abaf3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d9b17d-67659db75d-fl9f9"
Sep 17 14:11:34.771549 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:34.771509    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rfxq\" (UniqueName: \"kubernetes.io/projected/87759bfd-b7d8-45e0-b5f0-8d5c081abaf3-kube-api-access-2rfxq\") pod \"payload-pre-processing-e2e-worker-w0-d9b17d-67659db75d-fl9f9\" (UID: \"87759bfd-b7d8-45e0-b5f0-8d5c081abaf3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d9b17d-67659db75d-fl9f9"
Sep 17 14:11:34.771723 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:34.771611    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87759bfd-b7d8-45e0-b5f0-8d5c081abaf3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-d9b17d-67659db75d-fl9f9\" (UID: \"87759bfd-b7d8-45e0-b5f0-8d5c081abaf3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d9b17d-67659db75d-fl9f9"
Sep 17 14:11:34.772334 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:34.772314    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87759bfd-b7d8-45e0-b5f0-8d5c081abaf3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-d9b17d-67659db75d-fl9f9\" (UID: \"87759bfd-b7d8-45e0-b5f0-8d5c081abaf3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d9b17d-67659db75d-fl9f9"
Sep 17 14:11:34.778801 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:34.778779    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rfxq\" (UniqueName: \"kubernetes.io/projected/87759bfd-b7d8-45e0-b5f0-8d5c081abaf3-kube-api-access-2rfxq\") pod \"payload-pre-processing-e2e-worker-w0-d9b17d-67659db75d-fl9f9\" (UID: \"87759bfd-b7d8-45e0-b5f0-8d5c081abaf3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d9b17d-67659db75d-fl9f9"
Sep 17 14:11:34.852980 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:34.852944    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d9b17d-67659db75d-fl9f9"
Sep 17 14:11:34.931365 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:34.931274    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:35.005612 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:35.005580    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:35.134763 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:35.134732    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-d9b17d-67659db75d-fl9f9"]
Sep 17 14:11:35.138204 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:11:35.138175    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87759bfd_b7d8_45e0_b5f0_8d5c081abaf3.slice/crio-8c57355051f3cb23417bde5b45d9309eb426cc727e63a62589e1180199fce48f WatchSource:0}: Error finding container 8c57355051f3cb23417bde5b45d9309eb426cc727e63a62589e1180199fce48f: Status 404 returned error can't find the container with id 8c57355051f3cb23417bde5b45d9309eb426cc727e63a62589e1180199fce48f
Sep 17 14:11:35.231828 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:35.231744    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d9b17d-67659db75d-fl9f9" event={"ID":"87759bfd-b7d8-45e0-b5f0-8d5c081abaf3","Type":"ContainerStarted","Data":"8c57355051f3cb23417bde5b45d9309eb426cc727e63a62589e1180199fce48f"}
Sep 17 14:11:35.459379 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:35.459337    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:35.567835 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:35.567803    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:35.663499 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:35.663453    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:36.051973 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:36.051894    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:36.163700 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:36.163660    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:36.501447 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:36.501352    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:36.907216 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:36.907177    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:37.137822 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:37.137786    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:37.248316 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:37.248224    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d9b17d-67659db75d-fl9f9" event={"ID":"87759bfd-b7d8-45e0-b5f0-8d5c081abaf3","Type":"ContainerStarted","Data":"ffec03e3baa59e9d6940b2cac130304f9a877f41a38bb55a1bb5d3ee9ce0f2d5"}
Sep 17 14:11:37.248489 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:37.248335    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d9b17d-67659db75d-fl9f9"
Sep 17 14:11:37.269283 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:37.269227    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d9b17d-67659db75d-fl9f9" podStartSLOduration=1.415773982 podStartE2EDuration="3.269209382s" totalImagesPullingTime="1.8534354s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:11:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:11:37.265896278 +0000 UTC m=+966.208039924" watchObservedRunningTime="2026-09-17 14:11:37.269209382 +0000 UTC m=+966.211353050"
Sep 17 14:11:37.570081 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:37.570033    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:37.672946 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:37.672910    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:37.815390 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:37.815354    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:37.919755 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:37.919654    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:38.096085 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:38.096055    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd"]
Sep 17 14:11:38.100365 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:38.100345    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd"
Sep 17 14:11:38.102494 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:38.102444    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-1ee55ce9-openshift-default-dockercfg-xlnmv\""
Sep 17 14:11:38.110942 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:38.110921    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd"]
Sep 17 14:11:38.125156 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:38.125127    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:38.209120 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:38.209020    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5af3c4-6fdf-4e8d-931b-b4c67916fedc-workload-socket\") pod \"e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd\" (UID: \"7f5af3c4-6fdf-4e8d-931b-b4c67916fedc\") " pod="openshift-ingress/e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd"
Sep 17 14:11:38.209120 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:38.209086    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f5af3c4-6fdf-4e8d-931b-b4c67916fedc-istio-podinfo\") pod \"e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd\" (UID: \"7f5af3c4-6fdf-4e8d-931b-b4c67916fedc\") " pod="openshift-ingress/e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd"
Sep 17 14:11:38.209647 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:38.209136    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f5af3c4-6fdf-4e8d-931b-b4c67916fedc-istio-envoy\") pod \"e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd\" (UID: \"7f5af3c4-6fdf-4e8d-931b-b4c67916fedc\") " pod="openshift-ingress/e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd"
Sep 17 14:11:38.209647 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:38.209166    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqr86\" (UniqueName: \"kubernetes.io/projected/7f5af3c4-6fdf-4e8d-931b-b4c67916fedc-kube-api-access-pqr86\") pod \"e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd\" (UID: \"7f5af3c4-6fdf-4e8d-931b-b4c67916fedc\") " pod="openshift-ingress/e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd"
Sep 17 14:11:38.209647 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:38.209215    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f5af3c4-6fdf-4e8d-931b-b4c67916fedc-workload-certs\") pod \"e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd\" (UID: \"7f5af3c4-6fdf-4e8d-931b-b4c67916fedc\") " pod="openshift-ingress/e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd"
Sep 17 14:11:38.209647 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:38.209353    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5af3c4-6fdf-4e8d-931b-b4c67916fedc-credential-socket\") pod \"e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd\" (UID: \"7f5af3c4-6fdf-4e8d-931b-b4c67916fedc\") " pod="openshift-ingress/e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd"
Sep 17 14:11:38.209647 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:38.209411    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f5af3c4-6fdf-4e8d-931b-b4c67916fedc-istio-data\") pod \"e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd\" (UID: \"7f5af3c4-6fdf-4e8d-931b-b4c67916fedc\") " pod="openshift-ingress/e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd"
Sep 17 14:11:38.209647 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:38.209440    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f5af3c4-6fdf-4e8d-931b-b4c67916fedc-istiod-ca-cert\") pod \"e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd\" (UID: \"7f5af3c4-6fdf-4e8d-931b-b4c67916fedc\") " pod="openshift-ingress/e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd"
Sep 17 14:11:38.209647 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:38.209510    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f5af3c4-6fdf-4e8d-931b-b4c67916fedc-istio-token\") pod \"e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd\" (UID: \"7f5af3c4-6fdf-4e8d-931b-b4c67916fedc\") " pod="openshift-ingress/e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd"
Sep 17 14:11:38.310768 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:38.310736    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5af3c4-6fdf-4e8d-931b-b4c67916fedc-credential-socket\") pod \"e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd\" (UID: \"7f5af3c4-6fdf-4e8d-931b-b4c67916fedc\") " pod="openshift-ingress/e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd"
Sep 17 14:11:38.310973 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:38.310776    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f5af3c4-6fdf-4e8d-931b-b4c67916fedc-istio-data\") pod \"e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd\" (UID: \"7f5af3c4-6fdf-4e8d-931b-b4c67916fedc\") " pod="openshift-ingress/e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd"
Sep 17 14:11:38.310973 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:38.310796    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f5af3c4-6fdf-4e8d-931b-b4c67916fedc-istiod-ca-cert\") pod \"e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd\" (UID: \"7f5af3c4-6fdf-4e8d-931b-b4c67916fedc\") " pod="openshift-ingress/e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd"
Sep 17 14:11:38.310973 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:38.310845    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f5af3c4-6fdf-4e8d-931b-b4c67916fedc-istio-token\") pod \"e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd\" (UID: \"7f5af3c4-6fdf-4e8d-931b-b4c67916fedc\") " pod="openshift-ingress/e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd"
Sep 17 14:11:38.310973 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:38.310902    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5af3c4-6fdf-4e8d-931b-b4c67916fedc-workload-socket\") pod \"e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd\" (UID: \"7f5af3c4-6fdf-4e8d-931b-b4c67916fedc\") " pod="openshift-ingress/e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd"
Sep 17 14:11:38.310973 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:38.310935    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f5af3c4-6fdf-4e8d-931b-b4c67916fedc-istio-podinfo\") pod \"e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd\" (UID: \"7f5af3c4-6fdf-4e8d-931b-b4c67916fedc\") " pod="openshift-ingress/e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd"
Sep 17 14:11:38.310973 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:38.310975    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f5af3c4-6fdf-4e8d-931b-b4c67916fedc-istio-envoy\") pod \"e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd\" (UID: \"7f5af3c4-6fdf-4e8d-931b-b4c67916fedc\") " pod="openshift-ingress/e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd"
Sep 17 14:11:38.311309 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:38.311004    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqr86\" (UniqueName: \"kubernetes.io/projected/7f5af3c4-6fdf-4e8d-931b-b4c67916fedc-kube-api-access-pqr86\") pod \"e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd\" (UID: \"7f5af3c4-6fdf-4e8d-931b-b4c67916fedc\") " pod="openshift-ingress/e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd"
Sep 17 14:11:38.311309 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:38.311044    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f5af3c4-6fdf-4e8d-931b-b4c67916fedc-workload-certs\") pod \"e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd\" (UID: \"7f5af3c4-6fdf-4e8d-931b-b4c67916fedc\") " pod="openshift-ingress/e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd"
Sep 17 14:11:38.311309 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:38.311247    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5af3c4-6fdf-4e8d-931b-b4c67916fedc-credential-socket\") pod \"e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd\" (UID: \"7f5af3c4-6fdf-4e8d-931b-b4c67916fedc\") " pod="openshift-ingress/e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd"
Sep 17 14:11:38.311309 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:38.311284    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f5af3c4-6fdf-4e8d-931b-b4c67916fedc-istio-data\") pod \"e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd\" (UID: \"7f5af3c4-6fdf-4e8d-931b-b4c67916fedc\") " pod="openshift-ingress/e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd"
Sep 17 14:11:38.311823 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:38.311488    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5af3c4-6fdf-4e8d-931b-b4c67916fedc-workload-socket\") pod \"e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd\" (UID: \"7f5af3c4-6fdf-4e8d-931b-b4c67916fedc\") " pod="openshift-ingress/e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd"
Sep 17 14:11:38.312036 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:38.312004    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f5af3c4-6fdf-4e8d-931b-b4c67916fedc-workload-certs\") pod \"e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd\" (UID: \"7f5af3c4-6fdf-4e8d-931b-b4c67916fedc\") " pod="openshift-ingress/e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd"
Sep 17 14:11:38.312296 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:38.312192    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f5af3c4-6fdf-4e8d-931b-b4c67916fedc-istiod-ca-cert\") pod \"e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd\" (UID: \"7f5af3c4-6fdf-4e8d-931b-b4c67916fedc\") " pod="openshift-ingress/e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd"
Sep 17 14:11:38.319290 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:38.319264    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f5af3c4-6fdf-4e8d-931b-b4c67916fedc-istio-podinfo\") pod \"e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd\" (UID: \"7f5af3c4-6fdf-4e8d-931b-b4c67916fedc\") " pod="openshift-ingress/e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd"
Sep 17 14:11:38.320094 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:38.320056    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f5af3c4-6fdf-4e8d-931b-b4c67916fedc-istio-envoy\") pod \"e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd\" (UID: \"7f5af3c4-6fdf-4e8d-931b-b4c67916fedc\") " pod="openshift-ingress/e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd"
Sep 17 14:11:38.322397 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:38.322371    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f5af3c4-6fdf-4e8d-931b-b4c67916fedc-istio-token\") pod \"e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd\" (UID: \"7f5af3c4-6fdf-4e8d-931b-b4c67916fedc\") " pod="openshift-ingress/e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd"
Sep 17 14:11:38.322673 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:38.322643    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqr86\" (UniqueName: \"kubernetes.io/projected/7f5af3c4-6fdf-4e8d-931b-b4c67916fedc-kube-api-access-pqr86\") pod \"e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd\" (UID: \"7f5af3c4-6fdf-4e8d-931b-b4c67916fedc\") " pod="openshift-ingress/e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd"
Sep 17 14:11:38.411310 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:38.411277    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd"
Sep 17 14:11:38.432198 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:38.432162    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:38.693153 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:38.693125    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd"]
Sep 17 14:11:38.695306 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:11:38.695275    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f5af3c4_6fdf_4e8d_931b_b4c67916fedc.slice/crio-3603159b6d58b70509852a9acdc5be429690f84d0ce3cdcf650e54f407f0f822 WatchSource:0}: Error finding container 3603159b6d58b70509852a9acdc5be429690f84d0ce3cdcf650e54f407f0f822: Status 404 returned error can't find the container with id 3603159b6d58b70509852a9acdc5be429690f84d0ce3cdcf650e54f407f0f822
Sep 17 14:11:38.698164 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:38.698126    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:11:38.698254 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:38.698213    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:11:38.698295 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:38.698253    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:11:39.261100 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:39.261061    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd" event={"ID":"7f5af3c4-6fdf-4e8d-931b-b4c67916fedc","Type":"ContainerStarted","Data":"969189524cdb27e8e2e1b8ec09e9505851998e69536af154fd3d7f2b9ca4e211"}
Sep 17 14:11:39.261505 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:39.261124    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd" event={"ID":"7f5af3c4-6fdf-4e8d-931b-b4c67916fedc","Type":"ContainerStarted","Data":"3603159b6d58b70509852a9acdc5be429690f84d0ce3cdcf650e54f407f0f822"}
Sep 17 14:11:39.284786 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:39.284736    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd" podStartSLOduration=1.284720776 podStartE2EDuration="1.284720776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:11:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:11:39.282635232 +0000 UTC m=+968.224778879" watchObservedRunningTime="2026-09-17 14:11:39.284720776 +0000 UTC m=+968.226864419"
Sep 17 14:11:39.343318 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:39.343287    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:39.412063 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:39.412027    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd"
Sep 17 14:11:39.418770 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:39.418747    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd"
Sep 17 14:11:39.961307 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:39.961274    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:40.065100 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:40.065066    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:40.266901 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:40.266810    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd"
Sep 17 14:11:40.267884 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:40.267869    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd"
Sep 17 14:11:40.626339 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:40.626302    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:40.717997 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:40.717962    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:41.678353 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:41.678312    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:42.701098 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:42.701060    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:43.551095 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:43.551063    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:43.652407 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:43.652359    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:44.026079 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:44.025988    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:44.128778 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:44.128748    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:44.211271 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:44.210295    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:44.211987 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:44.211953    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1ee55ce9-66956858c5-4tt8t"]
Sep 17 14:11:44.216782 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:44.216758    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1ee55ce9-66956858c5-4tt8t"
Sep 17 14:11:44.220768 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:44.220228    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-1ee55ce9-dockercfg-rwg8d\""
Sep 17 14:11:44.220768 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:44.220539    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-1ee55ce9\""
Sep 17 14:11:44.224101 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:44.224075    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1ee55ce9-66956858c5-4tt8t"]
Sep 17 14:11:44.289377 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:44.289032    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c80794ff-c07f-45cd-bcc2-cedaa4438ce2-plugins-config-volume\") pod \"payload-processing-e2e-ait-1ee55ce9-66956858c5-4tt8t\" (UID: \"c80794ff-c07f-45cd-bcc2-cedaa4438ce2\") " pod="openshift-ingress/payload-processing-e2e-ait-1ee55ce9-66956858c5-4tt8t"
Sep 17 14:11:44.289377 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:44.289104    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8h6f\" (UniqueName: \"kubernetes.io/projected/c80794ff-c07f-45cd-bcc2-cedaa4438ce2-kube-api-access-p8h6f\") pod \"payload-processing-e2e-ait-1ee55ce9-66956858c5-4tt8t\" (UID: \"c80794ff-c07f-45cd-bcc2-cedaa4438ce2\") " pod="openshift-ingress/payload-processing-e2e-ait-1ee55ce9-66956858c5-4tt8t"
Sep 17 14:11:44.391487 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:44.391087    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c80794ff-c07f-45cd-bcc2-cedaa4438ce2-plugins-config-volume\") pod \"payload-processing-e2e-ait-1ee55ce9-66956858c5-4tt8t\" (UID: \"c80794ff-c07f-45cd-bcc2-cedaa4438ce2\") " pod="openshift-ingress/payload-processing-e2e-ait-1ee55ce9-66956858c5-4tt8t"
Sep 17 14:11:44.391487 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:44.391154    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8h6f\" (UniqueName: \"kubernetes.io/projected/c80794ff-c07f-45cd-bcc2-cedaa4438ce2-kube-api-access-p8h6f\") pod \"payload-processing-e2e-ait-1ee55ce9-66956858c5-4tt8t\" (UID: \"c80794ff-c07f-45cd-bcc2-cedaa4438ce2\") " pod="openshift-ingress/payload-processing-e2e-ait-1ee55ce9-66956858c5-4tt8t"
Sep 17 14:11:44.395484 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:44.392475    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c80794ff-c07f-45cd-bcc2-cedaa4438ce2-plugins-config-volume\") pod \"payload-processing-e2e-ait-1ee55ce9-66956858c5-4tt8t\" (UID: \"c80794ff-c07f-45cd-bcc2-cedaa4438ce2\") " pod="openshift-ingress/payload-processing-e2e-ait-1ee55ce9-66956858c5-4tt8t"
Sep 17 14:11:44.413561 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:44.413527    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8h6f\" (UniqueName: \"kubernetes.io/projected/c80794ff-c07f-45cd-bcc2-cedaa4438ce2-kube-api-access-p8h6f\") pod \"payload-processing-e2e-ait-1ee55ce9-66956858c5-4tt8t\" (UID: \"c80794ff-c07f-45cd-bcc2-cedaa4438ce2\") " pod="openshift-ingress/payload-processing-e2e-ait-1ee55ce9-66956858c5-4tt8t"
Sep 17 14:11:44.533333 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:44.533299    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1ee55ce9-66956858c5-4tt8t"
Sep 17 14:11:44.547135 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:44.547029    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:44.615517 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:44.615210    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:44.840646 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:44.840601    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:44.871862 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:44.869960    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1ee55ce9-66956858c5-4tt8t"]
Sep 17 14:11:44.873410 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:11:44.873372    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc80794ff_c07f_45cd_bcc2_cedaa4438ce2.slice/crio-d723740e0b553e152dd1a9d12d85247b91c1e70283d1ca00b323aba42d0cba04 WatchSource:0}: Error finding container d723740e0b553e152dd1a9d12d85247b91c1e70283d1ca00b323aba42d0cba04: Status 404 returned error can't find the container with id d723740e0b553e152dd1a9d12d85247b91c1e70283d1ca00b323aba42d0cba04
Sep 17 14:11:45.066123 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:45.065957    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:45.170305 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:45.170135    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:45.192118 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:45.192080    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-0458f0-6486b5497f-ddqdk"]
Sep 17 14:11:45.204533 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:45.204505    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-0458f0-6486b5497f-ddqdk"
Sep 17 14:11:45.210177 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:45.209169    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-0458f0-dockercfg-l6hsm\""
Sep 17 14:11:45.210177 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:45.209916    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-0458f0\""
Sep 17 14:11:45.232597 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:45.232543    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-0458f0-6486b5497f-ddqdk"]
Sep 17 14:11:45.316310 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:45.316066    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7smps\" (UniqueName: \"kubernetes.io/projected/f7e2e06f-693e-49f5-85ae-1f6768663911-kube-api-access-7smps\") pod \"payload-processing-e2e-autores-0458f0-6486b5497f-ddqdk\" (UID: \"f7e2e06f-693e-49f5-85ae-1f6768663911\") " pod="openshift-ingress/payload-processing-e2e-autores-0458f0-6486b5497f-ddqdk"
Sep 17 14:11:45.316310 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:45.316122    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7e2e06f-693e-49f5-85ae-1f6768663911-plugins-config-volume\") pod \"payload-processing-e2e-autores-0458f0-6486b5497f-ddqdk\" (UID: \"f7e2e06f-693e-49f5-85ae-1f6768663911\") " pod="openshift-ingress/payload-processing-e2e-autores-0458f0-6486b5497f-ddqdk"
Sep 17 14:11:45.332381 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:45.332344    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1ee55ce9-66956858c5-4tt8t" event={"ID":"c80794ff-c07f-45cd-bcc2-cedaa4438ce2","Type":"ContainerStarted","Data":"8a95864ac145ec0443d78d334d3e313f6506ac05ce383e9007ba5787521f0686"}
Sep 17 14:11:45.332691 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:45.332670    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1ee55ce9-66956858c5-4tt8t" event={"ID":"c80794ff-c07f-45cd-bcc2-cedaa4438ce2","Type":"ContainerStarted","Data":"d723740e0b553e152dd1a9d12d85247b91c1e70283d1ca00b323aba42d0cba04"}
Sep 17 14:11:45.333867 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:45.333844    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-1ee55ce9-66956858c5-4tt8t"
Sep 17 14:11:45.351568 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:45.351518    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-1ee55ce9-66956858c5-4tt8t" podStartSLOduration=1.351502249 podStartE2EDuration="1.351502249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:11:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:11:45.349748356 +0000 UTC m=+974.291892003" watchObservedRunningTime="2026-09-17 14:11:45.351502249 +0000 UTC m=+974.293645896"
Sep 17 14:11:45.417254 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:45.417204    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7smps\" (UniqueName: \"kubernetes.io/projected/f7e2e06f-693e-49f5-85ae-1f6768663911-kube-api-access-7smps\") pod \"payload-processing-e2e-autores-0458f0-6486b5497f-ddqdk\" (UID: \"f7e2e06f-693e-49f5-85ae-1f6768663911\") " pod="openshift-ingress/payload-processing-e2e-autores-0458f0-6486b5497f-ddqdk"
Sep 17 14:11:45.417555 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:45.417266    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7e2e06f-693e-49f5-85ae-1f6768663911-plugins-config-volume\") pod \"payload-processing-e2e-autores-0458f0-6486b5497f-ddqdk\" (UID: \"f7e2e06f-693e-49f5-85ae-1f6768663911\") " pod="openshift-ingress/payload-processing-e2e-autores-0458f0-6486b5497f-ddqdk"
Sep 17 14:11:45.418888 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:45.418859    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7e2e06f-693e-49f5-85ae-1f6768663911-plugins-config-volume\") pod \"payload-processing-e2e-autores-0458f0-6486b5497f-ddqdk\" (UID: \"f7e2e06f-693e-49f5-85ae-1f6768663911\") " pod="openshift-ingress/payload-processing-e2e-autores-0458f0-6486b5497f-ddqdk"
Sep 17 14:11:45.428533 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:45.427926    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7smps\" (UniqueName: \"kubernetes.io/projected/f7e2e06f-693e-49f5-85ae-1f6768663911-kube-api-access-7smps\") pod \"payload-processing-e2e-autores-0458f0-6486b5497f-ddqdk\" (UID: \"f7e2e06f-693e-49f5-85ae-1f6768663911\") " pod="openshift-ingress/payload-processing-e2e-autores-0458f0-6486b5497f-ddqdk"
Sep 17 14:11:45.529741 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:45.529697    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-0458f0-6486b5497f-ddqdk"
Sep 17 14:11:45.711095 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:45.710947    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:45.871121 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:45.871062    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-0458f0-6486b5497f-ddqdk"]
Sep 17 14:11:45.888939 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:11:45.888898    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7e2e06f_693e_49f5_85ae_1f6768663911.slice/crio-557d99b788dc46be8881d3790777d55841f52ff709d06c29c2366bd4d1436b48 WatchSource:0}: Error finding container 557d99b788dc46be8881d3790777d55841f52ff709d06c29c2366bd4d1436b48: Status 404 returned error can't find the container with id 557d99b788dc46be8881d3790777d55841f52ff709d06c29c2366bd4d1436b48
Sep 17 14:11:45.986583 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:45.986490    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:46.341028 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:46.340996    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-0458f0-6486b5497f-ddqdk" event={"ID":"f7e2e06f-693e-49f5-85ae-1f6768663911","Type":"ContainerStarted","Data":"5759e9db32b9d8a691adfda2cf3e99eec28a54eb2ba01dde283d8001729fdc53"}
Sep 17 14:11:46.341440 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:46.341041    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-0458f0-6486b5497f-ddqdk" event={"ID":"f7e2e06f-693e-49f5-85ae-1f6768663911","Type":"ContainerStarted","Data":"557d99b788dc46be8881d3790777d55841f52ff709d06c29c2366bd4d1436b48"}
Sep 17 14:11:46.341440 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:46.341075    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-0458f0-6486b5497f-ddqdk"
Sep 17 14:11:46.363413 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:46.363348    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-0458f0-6486b5497f-ddqdk" podStartSLOduration=1.363327766 podStartE2EDuration="1.363327766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:11:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:11:46.357822647 +0000 UTC m=+975.299966297" watchObservedRunningTime="2026-09-17 14:11:46.363327766 +0000 UTC m=+975.305471415"
Sep 17 14:11:46.426429 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:46.426395    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:46.533375 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:46.533257    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:46.625721 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:46.625611    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:47.680535 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:47.680440    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:48.255879 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:48.255849    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d9b17d-67659db75d-fl9f9"
Sep 17 14:11:48.546218 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:48.546115    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:48.650475 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:48.650372    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:48.744512 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:48.744474    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:48.845688 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:48.845651    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:48.945125 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:48.945089    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:49.460219 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:49.460188    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:50.190575 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:50.190540    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:50.716603 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:50.716576    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:50.980630 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:50.980523    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:51.158733 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:51.158692    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:51.251688 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:51.251604    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:51.356646 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:51.356617    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:51.378393 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:51.378363    2560 generic.go:356] "Generic (PLEG): container finished" podID="08fb3245-86c7-4cab-86d9-3b5177ca5101" containerID="3a6f61ffc5af81c362c8f2df3e6e6509b26ef1fe03f530db443d6b30a487fc7f" exitCode=0
Sep 17 14:11:51.378586 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:51.378443    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4" event={"ID":"08fb3245-86c7-4cab-86d9-3b5177ca5101","Type":"ContainerDied","Data":"3a6f61ffc5af81c362c8f2df3e6e6509b26ef1fe03f530db443d6b30a487fc7f"}
Sep 17 14:11:51.378960 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:51.378935    2560 scope.go:117] "RemoveContainer" containerID="3a6f61ffc5af81c362c8f2df3e6e6509b26ef1fe03f530db443d6b30a487fc7f"
Sep 17 14:11:51.380183 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:51.380146    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:11:51.380278 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:51.380256    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:11:51.380341 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:51.380309    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:11:51.380840 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:51.380822    2560 generic.go:356] "Generic (PLEG): container finished" podID="47668435-ed43-4f78-a71b-ceb972a66e25" containerID="cd85126c847fb7d5a7f45f2cc20ca0a4936ad72022acd9bca6c3598ed7c0d204" exitCode=0
Sep 17 14:11:51.380916 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:51.380898    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s" event={"ID":"47668435-ed43-4f78-a71b-ceb972a66e25","Type":"ContainerDied","Data":"cd85126c847fb7d5a7f45f2cc20ca0a4936ad72022acd9bca6c3598ed7c0d204"}
Sep 17 14:11:51.381244 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:51.381231    2560 scope.go:117] "RemoveContainer" containerID="cd85126c847fb7d5a7f45f2cc20ca0a4936ad72022acd9bca6c3598ed7c0d204"
Sep 17 14:11:51.382401 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:51.382371    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:11:51.382516 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:51.382453    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:11:51.382592 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:51.382523    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:11:51.530207 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:51.530047    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:51.662387 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:51.662344    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:51.860689 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:51.860648    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:52.196076 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:52.195993    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:52.296135 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:52.296093    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:52.388693 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:52.388660    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s" event={"ID":"47668435-ed43-4f78-a71b-ceb972a66e25","Type":"ContainerStarted","Data":"ad05498cd1fe227512c4c17430f86d321638ff7a9eda259aaedefed815f08517"}
Sep 17 14:11:52.390598 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:52.390570    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4" event={"ID":"08fb3245-86c7-4cab-86d9-3b5177ca5101","Type":"ContainerStarted","Data":"51d5b57fff878dd6b5d34d3dea78ff25bdd429ebecf3a0ebba0952328be5580b"}
Sep 17 14:11:53.021390 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:53.021361    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4"
Sep 17 14:11:53.021573 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:53.021489    2560 patch_prober.go:28] interesting pod/e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Sep 17 14:11:53.021573 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:53.021543    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4" podUID="08fb3245-86c7-4cab-86d9-3b5177ca5101" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Sep 17 14:11:53.099546 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:53.099509    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s"
Sep 17 14:11:53.099800 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:53.099770    2560 patch_prober.go:28] interesting pod/e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Sep 17 14:11:53.099889 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:53.099839    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s" podUID="47668435-ed43-4f78-a71b-ceb972a66e25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Sep 17 14:11:53.148083 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:53.148051    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:53.717353 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:53.717325    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:54.021006 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:54.020899    2560 patch_prober.go:28] interesting pod/e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Sep 17 14:11:54.021006 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:54.020949    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4" podUID="08fb3245-86c7-4cab-86d9-3b5177ca5101" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Sep 17 14:11:54.099252 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:54.099217    2560 patch_prober.go:28] interesting pod/e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Sep 17 14:11:54.099520 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:54.099492    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s" podUID="47668435-ed43-4f78-a71b-ceb972a66e25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Sep 17 14:11:54.273139 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:54.273042    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:55.020668 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:55.020636    2560 patch_prober.go:28] interesting pod/e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Sep 17 14:11:55.021070 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:55.020703    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4" podUID="08fb3245-86c7-4cab-86d9-3b5177ca5101" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Sep 17 14:11:55.082528 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:55.082486    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:55.099335 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:55.099294    2560 patch_prober.go:28] interesting pod/e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Sep 17 14:11:55.099535 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:55.099380    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s" podUID="47668435-ed43-4f78-a71b-ceb972a66e25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Sep 17 14:11:55.187361 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:55.187319    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:55.497402 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:55.497346    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:56.021006 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:56.020954    2560 patch_prober.go:28] interesting pod/e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Sep 17 14:11:56.021550 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:56.021043    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4" podUID="08fb3245-86c7-4cab-86d9-3b5177ca5101" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Sep 17 14:11:56.099506 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:56.099445    2560 patch_prober.go:28] interesting pod/e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Sep 17 14:11:56.099694 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:56.099533    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s" podUID="47668435-ed43-4f78-a71b-ceb972a66e25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Sep 17 14:11:56.602485 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:56.601515    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:57.021184 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:57.021081    2560 patch_prober.go:28] interesting pod/e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Sep 17 14:11:57.021829 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:57.021796    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4" podUID="08fb3245-86c7-4cab-86d9-3b5177ca5101" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Sep 17 14:11:57.102482 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:57.101920    2560 patch_prober.go:28] interesting pod/e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Sep 17 14:11:57.102482 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:57.102017    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s" podUID="47668435-ed43-4f78-a71b-ceb972a66e25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Sep 17 14:11:57.349397 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:57.349372    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-1ee55ce9-66956858c5-4tt8t"
Sep 17 14:11:57.362993 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:57.362962    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:57.446029 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:57.445997    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:57.550773 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:57.550737    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:57.907486 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:57.906855    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:58.027900 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:58.027864    2560 patch_prober.go:28] interesting pod/e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Sep 17 14:11:58.028443 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:58.028412    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4" podUID="08fb3245-86c7-4cab-86d9-3b5177ca5101" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Sep 17 14:11:58.041506 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:58.041300    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fe6ee5-6845887b87-fc7sw"]
Sep 17 14:11:58.046753 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:58.046729    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-fe6ee5-6845887b87-fc7sw"
Sep 17 14:11:58.049970 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:58.049953    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-fe6ee5\""
Sep 17 14:11:58.050131 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:58.049954    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-fe6ee5-dockercfg-5bm9r\""
Sep 17 14:11:58.055245 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:58.055218    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52gtt\" (UniqueName: \"kubernetes.io/projected/4697b97c-0029-421f-8016-c735369e3da4-kube-api-access-52gtt\") pod \"payload-processing-e2e-inf-b-fe6ee5-6845887b87-fc7sw\" (UID: \"4697b97c-0029-421f-8016-c735369e3da4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fe6ee5-6845887b87-fc7sw"
Sep 17 14:11:58.055370 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:58.055281    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4697b97c-0029-421f-8016-c735369e3da4-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-fe6ee5-6845887b87-fc7sw\" (UID: \"4697b97c-0029-421f-8016-c735369e3da4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fe6ee5-6845887b87-fc7sw"
Sep 17 14:11:58.055370 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:58.055305    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fe6ee5-6845887b87-fc7sw"]
Sep 17 14:11:58.101933 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:58.101897    2560 patch_prober.go:28] interesting pod/e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Sep 17 14:11:58.102191 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:58.102164    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s" podUID="47668435-ed43-4f78-a71b-ceb972a66e25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Sep 17 14:11:58.160545 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:58.159991    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52gtt\" (UniqueName: \"kubernetes.io/projected/4697b97c-0029-421f-8016-c735369e3da4-kube-api-access-52gtt\") pod \"payload-processing-e2e-inf-b-fe6ee5-6845887b87-fc7sw\" (UID: \"4697b97c-0029-421f-8016-c735369e3da4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fe6ee5-6845887b87-fc7sw"
Sep 17 14:11:58.160545 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:58.160067    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4697b97c-0029-421f-8016-c735369e3da4-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-fe6ee5-6845887b87-fc7sw\" (UID: \"4697b97c-0029-421f-8016-c735369e3da4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fe6ee5-6845887b87-fc7sw"
Sep 17 14:11:58.161488 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:58.161093    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4697b97c-0029-421f-8016-c735369e3da4-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-fe6ee5-6845887b87-fc7sw\" (UID: \"4697b97c-0029-421f-8016-c735369e3da4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fe6ee5-6845887b87-fc7sw"
Sep 17 14:11:58.185493 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:58.181369    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52gtt\" (UniqueName: \"kubernetes.io/projected/4697b97c-0029-421f-8016-c735369e3da4-kube-api-access-52gtt\") pod \"payload-processing-e2e-inf-b-fe6ee5-6845887b87-fc7sw\" (UID: \"4697b97c-0029-421f-8016-c735369e3da4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fe6ee5-6845887b87-fc7sw"
Sep 17 14:11:58.358722 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:58.358684    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-0458f0-6486b5497f-ddqdk"
Sep 17 14:11:58.362075 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:58.362047    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:58.370710 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:58.370683    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-fe6ee5-6845887b87-fc7sw"
Sep 17 14:11:58.467911 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:58.467837    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:58.681531 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:58.681491    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fe6ee5-6845887b87-fc7sw"]
Sep 17 14:11:58.687056 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:11:58.686974    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4697b97c_0029_421f_8016_c735369e3da4.slice/crio-ac9e3e71c3b63fe97fe007832a22a26def442fca2c5d987a673d647bdf611d32 WatchSource:0}: Error finding container ac9e3e71c3b63fe97fe007832a22a26def442fca2c5d987a673d647bdf611d32: Status 404 returned error can't find the container with id ac9e3e71c3b63fe97fe007832a22a26def442fca2c5d987a673d647bdf611d32
Sep 17 14:11:59.024181 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:59.024102    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4"
Sep 17 14:11:59.024397 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:59.024374    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4"
Sep 17 14:11:59.025166 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:59.025151    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4"
Sep 17 14:11:59.104126 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:59.104098    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s"
Sep 17 14:11:59.104986 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:59.104883    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s"
Sep 17 14:11:59.106144 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:59.106125    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s"
Sep 17 14:11:59.298970 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:59.298941    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:59.481993 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:59.481954    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-fe6ee5-6845887b87-fc7sw" event={"ID":"4697b97c-0029-421f-8016-c735369e3da4","Type":"ContainerStarted","Data":"ddb9e3d63b05dc857589355363f7a4748a4e901abcd755e3492c2569228a0fb8"}
Sep 17 14:11:59.482242 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:59.482222    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-fe6ee5-6845887b87-fc7sw" event={"ID":"4697b97c-0029-421f-8016-c735369e3da4","Type":"ContainerStarted","Data":"ac9e3e71c3b63fe97fe007832a22a26def442fca2c5d987a673d647bdf611d32"}
Sep 17 14:11:59.499399 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:59.499348    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-fe6ee5-6845887b87-fc7sw" podStartSLOduration=1.499331306 podStartE2EDuration="1.499331306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:11:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:11:59.498603433 +0000 UTC m=+988.440747079" watchObservedRunningTime="2026-09-17 14:11:59.499331306 +0000 UTC m=+988.441474956"
Sep 17 14:11:59.770502 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:11:59.770392    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:00.274215 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:00.274176    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:00.487235 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:00.487201    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-fe6ee5-6845887b87-fc7sw"
Sep 17 14:12:00.998456 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:00.998367    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ee55ce9-5wclx"]
Sep 17 14:12:01.003512 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:01.003492    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ee55ce9-5wclx"
Sep 17 14:12:01.006178 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:01.006119    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 14:12:01.006178 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:01.006170    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rp9ww\""
Sep 17 14:12:01.016373 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:01.016349    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ee55ce9-5wclx"]
Sep 17 14:12:01.097759 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:01.097725    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73185a80-e5a8-4183-85f2-370640de6903-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1ee55ce9-5wclx\" (UID: \"73185a80-e5a8-4183-85f2-370640de6903\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ee55ce9-5wclx"
Sep 17 14:12:01.198443 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:01.198412    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73185a80-e5a8-4183-85f2-370640de6903-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1ee55ce9-5wclx\" (UID: \"73185a80-e5a8-4183-85f2-370640de6903\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ee55ce9-5wclx"
Sep 17 14:12:01.199049 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:01.199030    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73185a80-e5a8-4183-85f2-370640de6903-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1ee55ce9-5wclx\" (UID: \"73185a80-e5a8-4183-85f2-370640de6903\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ee55ce9-5wclx"
Sep 17 14:12:01.291780 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:01.291684    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:01.322978 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:01.322944    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ee55ce9-5wclx"
Sep 17 14:12:01.609078 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:01.609046    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ee55ce9-5wclx"]
Sep 17 14:12:01.610247 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:12:01.610216    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73185a80_e5a8_4183_85f2_370640de6903.slice/crio-6428bc0ffb7a566fc0bd8b391fdced43461a6bcfd141c1d499ff20211ee8b47e WatchSource:0}: Error finding container 6428bc0ffb7a566fc0bd8b391fdced43461a6bcfd141c1d499ff20211ee8b47e: Status 404 returned error can't find the container with id 6428bc0ffb7a566fc0bd8b391fdced43461a6bcfd141c1d499ff20211ee8b47e
Sep 17 14:12:02.051424 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:02.051385    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:02.500203 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:02.500116    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ee55ce9-5wclx" event={"ID":"73185a80-e5a8-4183-85f2-370640de6903","Type":"ContainerStarted","Data":"6428bc0ffb7a566fc0bd8b391fdced43461a6bcfd141c1d499ff20211ee8b47e"}
Sep 17 14:12:02.718643 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:02.718600    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:02.943338 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:02.943294    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:03.836722 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:03.836688    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:04.514720 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:04.514692    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ee55ce9-5wclx" event={"ID":"73185a80-e5a8-4183-85f2-370640de6903","Type":"ContainerStarted","Data":"2b4ce32e0a2459468e224814337fb3ad52a8aee8b212a4a34b6f6915e2a6090e"}
Sep 17 14:12:04.516208 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:04.516187    2560 generic.go:356] "Generic (PLEG): container finished" podID="7f5af3c4-6fdf-4e8d-931b-b4c67916fedc" containerID="969189524cdb27e8e2e1b8ec09e9505851998e69536af154fd3d7f2b9ca4e211" exitCode=0
Sep 17 14:12:04.516304 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:04.516255    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd" event={"ID":"7f5af3c4-6fdf-4e8d-931b-b4c67916fedc","Type":"ContainerDied","Data":"969189524cdb27e8e2e1b8ec09e9505851998e69536af154fd3d7f2b9ca4e211"}
Sep 17 14:12:04.516623 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:04.516609    2560 scope.go:117] "RemoveContainer" containerID="969189524cdb27e8e2e1b8ec09e9505851998e69536af154fd3d7f2b9ca4e211"
Sep 17 14:12:04.517658 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:04.517632    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:12:04.517706 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:04.517698    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:12:04.517744 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:04.517727    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:12:04.533454 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:04.533434    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:04.539042 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:04.538998    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ee55ce9-5wclx" podStartSLOduration=2.01929692 podStartE2EDuration="4.538981151s" totalImagesPullingTime="2.519684231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:12:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:12:04.535336258 +0000 UTC m=+993.477479905" watchObservedRunningTime="2026-09-17 14:12:04.538981151 +0000 UTC m=+993.481125025"
Sep 17 14:12:05.331545 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:05.331513    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:05.523895 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:05.523853    2560 generic.go:356] "Generic (PLEG): container finished" podID="73185a80-e5a8-4183-85f2-370640de6903" containerID="2b4ce32e0a2459468e224814337fb3ad52a8aee8b212a4a34b6f6915e2a6090e" exitCode=0
Sep 17 14:12:05.524092 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:05.523937    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ee55ce9-5wclx" event={"ID":"73185a80-e5a8-4183-85f2-370640de6903","Type":"ContainerDied","Data":"2b4ce32e0a2459468e224814337fb3ad52a8aee8b212a4a34b6f6915e2a6090e"}
Sep 17 14:12:05.526014 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:05.525987    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd" event={"ID":"7f5af3c4-6fdf-4e8d-931b-b4c67916fedc","Type":"ContainerStarted","Data":"7ae0765d13785680e7e233be59c67068a1474be3a331379f51bf40458b59a3c2"}
Sep 17 14:12:06.100824 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:06.100789    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:06.412007 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:06.411928    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd"
Sep 17 14:12:06.413120 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:06.413095    2560 patch_prober.go:28] interesting pod/e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 17 14:12:06.413234 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:06.413163    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd" podUID="7f5af3c4-6fdf-4e8d-931b-b4c67916fedc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 17 14:12:06.709922 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:06.709898    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ee55ce9-5wclx"
Sep 17 14:12:06.754164 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:06.754134    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73185a80-e5a8-4183-85f2-370640de6903-maas-api-service-ca\") pod \"73185a80-e5a8-4183-85f2-370640de6903\" (UID: \"73185a80-e5a8-4183-85f2-370640de6903\") "
Sep 17 14:12:06.754525 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:06.754502    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73185a80-e5a8-4183-85f2-370640de6903-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "73185a80-e5a8-4183-85f2-370640de6903" (UID: "73185a80-e5a8-4183-85f2-370640de6903"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:12:06.855057 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:06.855026    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73185a80-e5a8-4183-85f2-370640de6903-maas-api-service-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:12:07.022613 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:07.022528    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:07.411677 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:07.411643    2560 patch_prober.go:28] interesting pod/e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 17 14:12:07.411847 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:07.411705    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd" podUID="7f5af3c4-6fdf-4e8d-931b-b4c67916fedc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 17 14:12:07.539786 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:07.539759    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ee55ce9-5wclx"
Sep 17 14:12:07.540164 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:07.539804    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ee55ce9-5wclx" event={"ID":"73185a80-e5a8-4183-85f2-370640de6903","Type":"ContainerDied","Data":"6428bc0ffb7a566fc0bd8b391fdced43461a6bcfd141c1d499ff20211ee8b47e"}
Sep 17 14:12:07.540164 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:07.539848    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6428bc0ffb7a566fc0bd8b391fdced43461a6bcfd141c1d499ff20211ee8b47e"
Sep 17 14:12:08.412430 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:08.412396    2560 patch_prober.go:28] interesting pod/e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 17 14:12:08.412629 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:08.412448    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd" podUID="7f5af3c4-6fdf-4e8d-931b-b4c67916fedc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 17 14:12:09.411694 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:09.411658    2560 patch_prober.go:28] interesting pod/e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 17 14:12:09.412080 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:09.411715    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd" podUID="7f5af3c4-6fdf-4e8d-931b-b4c67916fedc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 17 14:12:09.975645 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:09.975605    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:10.040550 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:10.040513    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:10.267563 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:10.267480    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd"
Sep 17 14:12:10.412308 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:10.412271    2560 patch_prober.go:28] interesting pod/e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 17 14:12:10.412837 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:10.412340    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd" podUID="7f5af3c4-6fdf-4e8d-931b-b4c67916fedc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 17 14:12:10.938517 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:10.938485    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:11.155725 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:11.155674    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:11.416170 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:11.416136    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd"
Sep 17 14:12:11.416978 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:11.416960    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd"
Sep 17 14:12:11.495636 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:11.495609    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-fe6ee5-6845887b87-fc7sw"
Sep 17 14:12:11.908417 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:11.908369    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1ee55ce9-66956858c5-4tt8t"]
Sep 17 14:12:11.908910 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:11.908879    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-1ee55ce9-66956858c5-4tt8t" podUID="c80794ff-c07f-45cd-bcc2-cedaa4438ce2" containerName="payload-processing" containerID="cri-o://8a95864ac145ec0443d78d334d3e313f6506ac05ce383e9007ba5787521f0686" gracePeriod=30
Sep 17 14:12:11.971289 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:11.971251    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:12.070541 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:12.070499    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:12.097522 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:12.097491    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ee55ce9-5wclx"]
Sep 17 14:12:12.100901 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:12.100868    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ee55ce9-5wclx"]
Sep 17 14:12:12.168856 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:12.167845    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1ee55ce9-66956858c5-4tt8t"
Sep 17 14:12:12.174023 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:12.173976    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:12.247369 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:12.247338    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd"]
Sep 17 14:12:12.314172 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:12.314139    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p8h6f\" (UniqueName: \"kubernetes.io/projected/c80794ff-c07f-45cd-bcc2-cedaa4438ce2-kube-api-access-p8h6f\") pod \"c80794ff-c07f-45cd-bcc2-cedaa4438ce2\" (UID: \"c80794ff-c07f-45cd-bcc2-cedaa4438ce2\") "
Sep 17 14:12:12.314565 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:12.314193    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c80794ff-c07f-45cd-bcc2-cedaa4438ce2-plugins-config-volume\") pod \"c80794ff-c07f-45cd-bcc2-cedaa4438ce2\" (UID: \"c80794ff-c07f-45cd-bcc2-cedaa4438ce2\") "
Sep 17 14:12:12.314737 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:12.314642    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c80794ff-c07f-45cd-bcc2-cedaa4438ce2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c80794ff-c07f-45cd-bcc2-cedaa4438ce2" (UID: "c80794ff-c07f-45cd-bcc2-cedaa4438ce2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:12:12.316112 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:12.316087    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:12.316937 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:12.316911    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c80794ff-c07f-45cd-bcc2-cedaa4438ce2-kube-api-access-p8h6f" (OuterVolumeSpecName: "kube-api-access-p8h6f") pod "c80794ff-c07f-45cd-bcc2-cedaa4438ce2" (UID: "c80794ff-c07f-45cd-bcc2-cedaa4438ce2"). InnerVolumeSpecName "kube-api-access-p8h6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:12:12.415519 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:12.415489    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8h6f\" (UniqueName: \"kubernetes.io/projected/c80794ff-c07f-45cd-bcc2-cedaa4438ce2-kube-api-access-p8h6f\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:12:12.415519 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:12.415519    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c80794ff-c07f-45cd-bcc2-cedaa4438ce2-plugins-config-volume\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:12:12.551216 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:12.551169    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:12.569941 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:12.569915    2560 generic.go:356] "Generic (PLEG): container finished" podID="c80794ff-c07f-45cd-bcc2-cedaa4438ce2" containerID="8a95864ac145ec0443d78d334d3e313f6506ac05ce383e9007ba5787521f0686" exitCode=0
Sep 17 14:12:12.570073 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:12.569941    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1ee55ce9-66956858c5-4tt8t" event={"ID":"c80794ff-c07f-45cd-bcc2-cedaa4438ce2","Type":"ContainerDied","Data":"8a95864ac145ec0443d78d334d3e313f6506ac05ce383e9007ba5787521f0686"}
Sep 17 14:12:12.570073 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:12.569973    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1ee55ce9-66956858c5-4tt8t"
Sep 17 14:12:12.570073 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:12.569991    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1ee55ce9-66956858c5-4tt8t" event={"ID":"c80794ff-c07f-45cd-bcc2-cedaa4438ce2","Type":"ContainerDied","Data":"d723740e0b553e152dd1a9d12d85247b91c1e70283d1ca00b323aba42d0cba04"}
Sep 17 14:12:12.570073 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:12.570016    2560 scope.go:117] "RemoveContainer" containerID="8a95864ac145ec0443d78d334d3e313f6506ac05ce383e9007ba5787521f0686"
Sep 17 14:12:12.570288 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:12.570263    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd" podUID="7f5af3c4-6fdf-4e8d-931b-b4c67916fedc" containerName="istio-proxy" containerID="cri-o://7ae0765d13785680e7e233be59c67068a1474be3a331379f51bf40458b59a3c2" gracePeriod=30
Sep 17 14:12:12.580029 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:12.580011    2560 scope.go:117] "RemoveContainer" containerID="8a95864ac145ec0443d78d334d3e313f6506ac05ce383e9007ba5787521f0686"
Sep 17 14:12:12.580303 ip-10-0-129-158 kubenswrapper[2560]: E0917 14:12:12.580285    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a95864ac145ec0443d78d334d3e313f6506ac05ce383e9007ba5787521f0686\": container with ID starting with 8a95864ac145ec0443d78d334d3e313f6506ac05ce383e9007ba5787521f0686 not found: ID does not exist" containerID="8a95864ac145ec0443d78d334d3e313f6506ac05ce383e9007ba5787521f0686"
Sep 17 14:12:12.580386 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:12.580316    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a95864ac145ec0443d78d334d3e313f6506ac05ce383e9007ba5787521f0686"} err="failed to get container status \"8a95864ac145ec0443d78d334d3e313f6506ac05ce383e9007ba5787521f0686\": rpc error: code = NotFound desc = could not find container \"8a95864ac145ec0443d78d334d3e313f6506ac05ce383e9007ba5787521f0686\": container with ID starting with 8a95864ac145ec0443d78d334d3e313f6506ac05ce383e9007ba5787521f0686 not found: ID does not exist"
Sep 17 14:12:12.613062 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:12.613038    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1ee55ce9-66956858c5-4tt8t"]
Sep 17 14:12:12.619226 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:12.619194    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1ee55ce9-66956858c5-4tt8t"]
Sep 17 14:12:12.740793 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:12.740765    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:12.838088 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:12.838016    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:12.961256 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:12.961225    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:13.251409 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:13.251336    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672"]
Sep 17 14:12:13.252056 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:13.252036    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73185a80-e5a8-4183-85f2-370640de6903" containerName="revoke-keys"
Sep 17 14:12:13.252104 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:13.252063    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="73185a80-e5a8-4183-85f2-370640de6903" containerName="revoke-keys"
Sep 17 14:12:13.252141 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:13.252101    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c80794ff-c07f-45cd-bcc2-cedaa4438ce2" containerName="payload-processing"
Sep 17 14:12:13.252141 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:13.252111    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c80794ff-c07f-45cd-bcc2-cedaa4438ce2" containerName="payload-processing"
Sep 17 14:12:13.252253 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:13.252234    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="73185a80-e5a8-4183-85f2-370640de6903" containerName="revoke-keys"
Sep 17 14:12:13.252381 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:13.252260    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="c80794ff-c07f-45cd-bcc2-cedaa4438ce2" containerName="payload-processing"
Sep 17 14:12:13.256665 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:13.256648    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672"
Sep 17 14:12:13.259430 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:13.259412    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-dd410a86-gw-openshift-default-dockercfg-mpvc9\""
Sep 17 14:12:13.267095 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:13.267075    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672"]
Sep 17 14:12:13.276297 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:13.276273    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:13.326978 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:13.326953    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fq8s\" (UniqueName: \"kubernetes.io/projected/9276b8fd-fd62-46b3-bdd5-612a7a39b59b-kube-api-access-7fq8s\") pod \"e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672\" (UID: \"9276b8fd-fd62-46b3-bdd5-612a7a39b59b\") " pod="openshift-ingress/e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672"
Sep 17 14:12:13.327101 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:13.326989    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9276b8fd-fd62-46b3-bdd5-612a7a39b59b-workload-certs\") pod \"e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672\" (UID: \"9276b8fd-fd62-46b3-bdd5-612a7a39b59b\") " pod="openshift-ingress/e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672"
Sep 17 14:12:13.327101 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:13.327007    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9276b8fd-fd62-46b3-bdd5-612a7a39b59b-istio-token\") pod \"e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672\" (UID: \"9276b8fd-fd62-46b3-bdd5-612a7a39b59b\") " pod="openshift-ingress/e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672"
Sep 17 14:12:13.327174 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:13.327102    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9276b8fd-fd62-46b3-bdd5-612a7a39b59b-workload-socket\") pod \"e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672\" (UID: \"9276b8fd-fd62-46b3-bdd5-612a7a39b59b\") " pod="openshift-ingress/e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672"
Sep 17 14:12:13.327174 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:13.327144    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9276b8fd-fd62-46b3-bdd5-612a7a39b59b-credential-socket\") pod \"e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672\" (UID: \"9276b8fd-fd62-46b3-bdd5-612a7a39b59b\") " pod="openshift-ingress/e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672"
Sep 17 14:12:13.327249 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:13.327179    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9276b8fd-fd62-46b3-bdd5-612a7a39b59b-istio-envoy\") pod \"e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672\" (UID: \"9276b8fd-fd62-46b3-bdd5-612a7a39b59b\") " pod="openshift-ingress/e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672"
Sep 17 14:12:13.327249 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:13.327202    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9276b8fd-fd62-46b3-bdd5-612a7a39b59b-istio-data\") pod \"e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672\" (UID: \"9276b8fd-fd62-46b3-bdd5-612a7a39b59b\") " pod="openshift-ingress/e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672"
Sep 17 14:12:13.327249 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:13.327218    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9276b8fd-fd62-46b3-bdd5-612a7a39b59b-istio-podinfo\") pod \"e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672\" (UID: \"9276b8fd-fd62-46b3-bdd5-612a7a39b59b\") " pod="openshift-ingress/e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672"
Sep 17 14:12:13.327342 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:13.327246    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9276b8fd-fd62-46b3-bdd5-612a7a39b59b-istiod-ca-cert\") pod \"e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672\" (UID: \"9276b8fd-fd62-46b3-bdd5-612a7a39b59b\") " pod="openshift-ingress/e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672"
Sep 17 14:12:13.428537 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:13.428502    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9276b8fd-fd62-46b3-bdd5-612a7a39b59b-istio-envoy\") pod \"e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672\" (UID: \"9276b8fd-fd62-46b3-bdd5-612a7a39b59b\") " pod="openshift-ingress/e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672"
Sep 17 14:12:13.428537 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:13.428541    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9276b8fd-fd62-46b3-bdd5-612a7a39b59b-istio-data\") pod \"e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672\" (UID: \"9276b8fd-fd62-46b3-bdd5-612a7a39b59b\") " pod="openshift-ingress/e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672"
Sep 17 14:12:13.428790 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:13.428559    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9276b8fd-fd62-46b3-bdd5-612a7a39b59b-istio-podinfo\") pod \"e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672\" (UID: \"9276b8fd-fd62-46b3-bdd5-612a7a39b59b\") " pod="openshift-ingress/e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672"
Sep 17 14:12:13.428790 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:13.428582    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9276b8fd-fd62-46b3-bdd5-612a7a39b59b-istiod-ca-cert\") pod \"e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672\" (UID: \"9276b8fd-fd62-46b3-bdd5-612a7a39b59b\") " pod="openshift-ingress/e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672"
Sep 17 14:12:13.428790 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:13.428651    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fq8s\" (UniqueName: \"kubernetes.io/projected/9276b8fd-fd62-46b3-bdd5-612a7a39b59b-kube-api-access-7fq8s\") pod \"e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672\" (UID: \"9276b8fd-fd62-46b3-bdd5-612a7a39b59b\") " pod="openshift-ingress/e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672"
Sep 17 14:12:13.428790 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:13.428681    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9276b8fd-fd62-46b3-bdd5-612a7a39b59b-workload-certs\") pod \"e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672\" (UID: \"9276b8fd-fd62-46b3-bdd5-612a7a39b59b\") " pod="openshift-ingress/e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672"
Sep 17 14:12:13.428790 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:13.428705    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9276b8fd-fd62-46b3-bdd5-612a7a39b59b-istio-token\") pod \"e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672\" (UID: \"9276b8fd-fd62-46b3-bdd5-612a7a39b59b\") " pod="openshift-ingress/e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672"
Sep 17 14:12:13.428790 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:13.428783    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9276b8fd-fd62-46b3-bdd5-612a7a39b59b-workload-socket\") pod \"e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672\" (UID: \"9276b8fd-fd62-46b3-bdd5-612a7a39b59b\") " pod="openshift-ingress/e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672"
Sep 17 14:12:13.429107 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:13.428836    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9276b8fd-fd62-46b3-bdd5-612a7a39b59b-credential-socket\") pod \"e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672\" (UID: \"9276b8fd-fd62-46b3-bdd5-612a7a39b59b\") " pod="openshift-ingress/e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672"
Sep 17 14:12:13.429107 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:13.428980    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9276b8fd-fd62-46b3-bdd5-612a7a39b59b-istio-data\") pod \"e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672\" (UID: \"9276b8fd-fd62-46b3-bdd5-612a7a39b59b\") " pod="openshift-ingress/e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672"
Sep 17 14:12:13.429107 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:13.429020    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9276b8fd-fd62-46b3-bdd5-612a7a39b59b-workload-certs\") pod \"e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672\" (UID: \"9276b8fd-fd62-46b3-bdd5-612a7a39b59b\") " pod="openshift-ingress/e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672"
Sep 17 14:12:13.429283 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:13.429259    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9276b8fd-fd62-46b3-bdd5-612a7a39b59b-workload-socket\") pod \"e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672\" (UID: \"9276b8fd-fd62-46b3-bdd5-612a7a39b59b\") " pod="openshift-ingress/e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672"
Sep 17 14:12:13.429344 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:13.429321    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9276b8fd-fd62-46b3-bdd5-612a7a39b59b-credential-socket\") pod \"e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672\" (UID: \"9276b8fd-fd62-46b3-bdd5-612a7a39b59b\") " pod="openshift-ingress/e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672"
Sep 17 14:12:13.429621 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:13.429600    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9276b8fd-fd62-46b3-bdd5-612a7a39b59b-istiod-ca-cert\") pod \"e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672\" (UID: \"9276b8fd-fd62-46b3-bdd5-612a7a39b59b\") " pod="openshift-ingress/e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672"
Sep 17 14:12:13.431040 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:13.431020    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9276b8fd-fd62-46b3-bdd5-612a7a39b59b-istio-envoy\") pod \"e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672\" (UID: \"9276b8fd-fd62-46b3-bdd5-612a7a39b59b\") " pod="openshift-ingress/e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672"
Sep 17 14:12:13.431186 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:13.431170    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9276b8fd-fd62-46b3-bdd5-612a7a39b59b-istio-podinfo\") pod \"e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672\" (UID: \"9276b8fd-fd62-46b3-bdd5-612a7a39b59b\") " pod="openshift-ingress/e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672"
Sep 17 14:12:13.442675 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:13.442653    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fq8s\" (UniqueName: \"kubernetes.io/projected/9276b8fd-fd62-46b3-bdd5-612a7a39b59b-kube-api-access-7fq8s\") pod \"e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672\" (UID: \"9276b8fd-fd62-46b3-bdd5-612a7a39b59b\") " pod="openshift-ingress/e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672"
Sep 17 14:12:13.443104 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:13.443087    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9276b8fd-fd62-46b3-bdd5-612a7a39b59b-istio-token\") pod \"e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672\" (UID: \"9276b8fd-fd62-46b3-bdd5-612a7a39b59b\") " pod="openshift-ingress/e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672"
Sep 17 14:12:13.488338 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:13.488304    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:13.568612 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:13.568593    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672"
Sep 17 14:12:13.569937 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:13.569915    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73185a80-e5a8-4183-85f2-370640de6903" path="/var/lib/kubelet/pods/73185a80-e5a8-4183-85f2-370640de6903/volumes"
Sep 17 14:12:13.570294 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:13.570281    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c80794ff-c07f-45cd-bcc2-cedaa4438ce2" path="/var/lib/kubelet/pods/c80794ff-c07f-45cd-bcc2-cedaa4438ce2/volumes"
Sep 17 14:12:13.670627 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:13.670572    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:13.849250 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:13.849225    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672"]
Sep 17 14:12:13.849961 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:12:13.849937    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9276b8fd_fd62_46b3_bdd5_612a7a39b59b.slice/crio-868f87a7e81da39960bacdf45f3434288d6ea36581b77f9da29da2d43cb29e1d WatchSource:0}: Error finding container 868f87a7e81da39960bacdf45f3434288d6ea36581b77f9da29da2d43cb29e1d: Status 404 returned error can't find the container with id 868f87a7e81da39960bacdf45f3434288d6ea36581b77f9da29da2d43cb29e1d
Sep 17 14:12:13.852271 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:13.852242    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:12:13.852336 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:13.852300    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:12:13.852336 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:13.852324    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:12:14.540395 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:14.540356    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:14.584493 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:14.584446    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672" event={"ID":"9276b8fd-fd62-46b3-bdd5-612a7a39b59b","Type":"ContainerStarted","Data":"d05cc22ffddc4253c9e6cd0c3cc987867a3a4d2be00ee503f627cde7c8fbc0f4"}
Sep 17 14:12:14.584851 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:14.584514    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672" event={"ID":"9276b8fd-fd62-46b3-bdd5-612a7a39b59b","Type":"ContainerStarted","Data":"868f87a7e81da39960bacdf45f3434288d6ea36581b77f9da29da2d43cb29e1d"}
Sep 17 14:12:14.610161 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:14.610112    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672" podStartSLOduration=1.6100972709999999 podStartE2EDuration="1.610097271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:12:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:12:14.607889602 +0000 UTC m=+1003.550033247" watchObservedRunningTime="2026-09-17 14:12:14.610097271 +0000 UTC m=+1003.552240916"
Sep 17 14:12:15.472991 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:15.472951    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:15.569889 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:15.569860    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672"
Sep 17 14:12:15.574689 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:15.574666    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672"
Sep 17 14:12:15.592859 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:15.592838    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672"
Sep 17 14:12:15.593473 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:15.593444    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672"
Sep 17 14:12:16.105130 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:16.105099    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:16.205974 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:16.205941    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:16.416603 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:16.416513    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:17.231647 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:17.231611    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:17.819275 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:17.819251    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd"
Sep 17 14:12:17.971948 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:17.971839    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f5af3c4-6fdf-4e8d-931b-b4c67916fedc-istio-envoy\") pod \"7f5af3c4-6fdf-4e8d-931b-b4c67916fedc\" (UID: \"7f5af3c4-6fdf-4e8d-931b-b4c67916fedc\") "
Sep 17 14:12:17.971948 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:17.971940    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f5af3c4-6fdf-4e8d-931b-b4c67916fedc-istio-token\") pod \"7f5af3c4-6fdf-4e8d-931b-b4c67916fedc\" (UID: \"7f5af3c4-6fdf-4e8d-931b-b4c67916fedc\") "
Sep 17 14:12:17.972195 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:17.971982    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5af3c4-6fdf-4e8d-931b-b4c67916fedc-workload-socket\") pod \"7f5af3c4-6fdf-4e8d-931b-b4c67916fedc\" (UID: \"7f5af3c4-6fdf-4e8d-931b-b4c67916fedc\") "
Sep 17 14:12:17.972195 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:17.972043    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f5af3c4-6fdf-4e8d-931b-b4c67916fedc-istio-data\") pod \"7f5af3c4-6fdf-4e8d-931b-b4c67916fedc\" (UID: \"7f5af3c4-6fdf-4e8d-931b-b4c67916fedc\") "
Sep 17 14:12:17.972195 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:17.972077    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f5af3c4-6fdf-4e8d-931b-b4c67916fedc-istiod-ca-cert\") pod \"7f5af3c4-6fdf-4e8d-931b-b4c67916fedc\" (UID: \"7f5af3c4-6fdf-4e8d-931b-b4c67916fedc\") "
Sep 17 14:12:17.972195 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:17.972106    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f5af3c4-6fdf-4e8d-931b-b4c67916fedc-istio-podinfo\") pod \"7f5af3c4-6fdf-4e8d-931b-b4c67916fedc\" (UID: \"7f5af3c4-6fdf-4e8d-931b-b4c67916fedc\") "
Sep 17 14:12:17.972195 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:17.972175    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pqr86\" (UniqueName: \"kubernetes.io/projected/7f5af3c4-6fdf-4e8d-931b-b4c67916fedc-kube-api-access-pqr86\") pod \"7f5af3c4-6fdf-4e8d-931b-b4c67916fedc\" (UID: \"7f5af3c4-6fdf-4e8d-931b-b4c67916fedc\") "
Sep 17 14:12:17.972455 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:17.972215    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5af3c4-6fdf-4e8d-931b-b4c67916fedc-credential-socket\") pod \"7f5af3c4-6fdf-4e8d-931b-b4c67916fedc\" (UID: \"7f5af3c4-6fdf-4e8d-931b-b4c67916fedc\") "
Sep 17 14:12:17.972455 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:17.972247    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f5af3c4-6fdf-4e8d-931b-b4c67916fedc-workload-certs\") pod \"7f5af3c4-6fdf-4e8d-931b-b4c67916fedc\" (UID: \"7f5af3c4-6fdf-4e8d-931b-b4c67916fedc\") "
Sep 17 14:12:17.972455 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:17.972348    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f5af3c4-6fdf-4e8d-931b-b4c67916fedc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7f5af3c4-6fdf-4e8d-931b-b4c67916fedc" (UID: "7f5af3c4-6fdf-4e8d-931b-b4c67916fedc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:12:17.972642 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:17.972581    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f5af3c4-6fdf-4e8d-931b-b4c67916fedc-istio-data" (OuterVolumeSpecName: "istio-data") pod "7f5af3c4-6fdf-4e8d-931b-b4c67916fedc" (UID: "7f5af3c4-6fdf-4e8d-931b-b4c67916fedc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:12:17.972701 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:17.972656    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f5af3c4-6fdf-4e8d-931b-b4c67916fedc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7f5af3c4-6fdf-4e8d-931b-b4c67916fedc" (UID: "7f5af3c4-6fdf-4e8d-931b-b4c67916fedc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:12:17.972764 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:17.972744    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5af3c4-6fdf-4e8d-931b-b4c67916fedc-workload-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:12:17.972811 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:17.972762    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f5af3c4-6fdf-4e8d-931b-b4c67916fedc-istio-data\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:12:17.972811 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:17.972776    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f5af3c4-6fdf-4e8d-931b-b4c67916fedc-istiod-ca-cert\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:12:17.973191 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:17.973115    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f5af3c4-6fdf-4e8d-931b-b4c67916fedc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7f5af3c4-6fdf-4e8d-931b-b4c67916fedc" (UID: "7f5af3c4-6fdf-4e8d-931b-b4c67916fedc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:12:17.973510 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:17.973413    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f5af3c4-6fdf-4e8d-931b-b4c67916fedc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7f5af3c4-6fdf-4e8d-931b-b4c67916fedc" (UID: "7f5af3c4-6fdf-4e8d-931b-b4c67916fedc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:12:17.975197 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:17.975132    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f5af3c4-6fdf-4e8d-931b-b4c67916fedc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7f5af3c4-6fdf-4e8d-931b-b4c67916fedc" (UID: "7f5af3c4-6fdf-4e8d-931b-b4c67916fedc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 14:12:17.975197 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:17.975151    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f5af3c4-6fdf-4e8d-931b-b4c67916fedc-kube-api-access-pqr86" (OuterVolumeSpecName: "kube-api-access-pqr86") pod "7f5af3c4-6fdf-4e8d-931b-b4c67916fedc" (UID: "7f5af3c4-6fdf-4e8d-931b-b4c67916fedc"). InnerVolumeSpecName "kube-api-access-pqr86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:12:17.975197 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:17.975151    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f5af3c4-6fdf-4e8d-931b-b4c67916fedc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7f5af3c4-6fdf-4e8d-931b-b4c67916fedc" (UID: "7f5af3c4-6fdf-4e8d-931b-b4c67916fedc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:12:17.975396 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:17.975377    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f5af3c4-6fdf-4e8d-931b-b4c67916fedc-istio-token" (OuterVolumeSpecName: "istio-token") pod "7f5af3c4-6fdf-4e8d-931b-b4c67916fedc" (UID: "7f5af3c4-6fdf-4e8d-931b-b4c67916fedc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:12:18.073854 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:18.073824    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f5af3c4-6fdf-4e8d-931b-b4c67916fedc-istio-podinfo\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:12:18.073854 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:18.073848    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqr86\" (UniqueName: \"kubernetes.io/projected/7f5af3c4-6fdf-4e8d-931b-b4c67916fedc-kube-api-access-pqr86\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:12:18.074030 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:18.073863    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5af3c4-6fdf-4e8d-931b-b4c67916fedc-credential-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:12:18.074030 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:18.073876    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f5af3c4-6fdf-4e8d-931b-b4c67916fedc-workload-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:12:18.074030 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:18.073886    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f5af3c4-6fdf-4e8d-931b-b4c67916fedc-istio-envoy\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:12:18.074030 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:18.073894    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f5af3c4-6fdf-4e8d-931b-b4c67916fedc-istio-token\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:12:18.480028 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:18.479990    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:18.618757 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:18.618722    2560 generic.go:356] "Generic (PLEG): container finished" podID="7f5af3c4-6fdf-4e8d-931b-b4c67916fedc" containerID="7ae0765d13785680e7e233be59c67068a1474be3a331379f51bf40458b59a3c2" exitCode=0
Sep 17 14:12:18.618918 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:18.618778    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd"
Sep 17 14:12:18.618918 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:18.618770    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd" event={"ID":"7f5af3c4-6fdf-4e8d-931b-b4c67916fedc","Type":"ContainerDied","Data":"7ae0765d13785680e7e233be59c67068a1474be3a331379f51bf40458b59a3c2"}
Sep 17 14:12:18.618918 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:18.618895    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd" event={"ID":"7f5af3c4-6fdf-4e8d-931b-b4c67916fedc","Type":"ContainerDied","Data":"3603159b6d58b70509852a9acdc5be429690f84d0ce3cdcf650e54f407f0f822"}
Sep 17 14:12:18.619020 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:18.618919    2560 scope.go:117] "RemoveContainer" containerID="7ae0765d13785680e7e233be59c67068a1474be3a331379f51bf40458b59a3c2"
Sep 17 14:12:18.628803 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:18.628783    2560 scope.go:117] "RemoveContainer" containerID="969189524cdb27e8e2e1b8ec09e9505851998e69536af154fd3d7f2b9ca4e211"
Sep 17 14:12:18.637244 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:18.637223    2560 scope.go:117] "RemoveContainer" containerID="7ae0765d13785680e7e233be59c67068a1474be3a331379f51bf40458b59a3c2"
Sep 17 14:12:18.637528 ip-10-0-129-158 kubenswrapper[2560]: E0917 14:12:18.637508    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ae0765d13785680e7e233be59c67068a1474be3a331379f51bf40458b59a3c2\": container with ID starting with 7ae0765d13785680e7e233be59c67068a1474be3a331379f51bf40458b59a3c2 not found: ID does not exist" containerID="7ae0765d13785680e7e233be59c67068a1474be3a331379f51bf40458b59a3c2"
Sep 17 14:12:18.637591 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:18.637537    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ae0765d13785680e7e233be59c67068a1474be3a331379f51bf40458b59a3c2"} err="failed to get container status \"7ae0765d13785680e7e233be59c67068a1474be3a331379f51bf40458b59a3c2\": rpc error: code = NotFound desc = could not find container \"7ae0765d13785680e7e233be59c67068a1474be3a331379f51bf40458b59a3c2\": container with ID starting with 7ae0765d13785680e7e233be59c67068a1474be3a331379f51bf40458b59a3c2 not found: ID does not exist"
Sep 17 14:12:18.637591 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:18.637552    2560 scope.go:117] "RemoveContainer" containerID="969189524cdb27e8e2e1b8ec09e9505851998e69536af154fd3d7f2b9ca4e211"
Sep 17 14:12:18.637768 ip-10-0-129-158 kubenswrapper[2560]: E0917 14:12:18.637754    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"969189524cdb27e8e2e1b8ec09e9505851998e69536af154fd3d7f2b9ca4e211\": container with ID starting with 969189524cdb27e8e2e1b8ec09e9505851998e69536af154fd3d7f2b9ca4e211 not found: ID does not exist" containerID="969189524cdb27e8e2e1b8ec09e9505851998e69536af154fd3d7f2b9ca4e211"
Sep 17 14:12:18.637810 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:18.637772    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"969189524cdb27e8e2e1b8ec09e9505851998e69536af154fd3d7f2b9ca4e211"} err="failed to get container status \"969189524cdb27e8e2e1b8ec09e9505851998e69536af154fd3d7f2b9ca4e211\": rpc error: code = NotFound desc = could not find container \"969189524cdb27e8e2e1b8ec09e9505851998e69536af154fd3d7f2b9ca4e211\": container with ID starting with 969189524cdb27e8e2e1b8ec09e9505851998e69536af154fd3d7f2b9ca4e211 not found: ID does not exist"
Sep 17 14:12:18.646709 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:18.646681    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd"]
Sep 17 14:12:18.653605 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:18.653583    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-1ee55ce9-openshift-default-774686748d-jzgdd"]
Sep 17 14:12:19.571020 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:19.570985    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f5af3c4-6fdf-4e8d-931b-b4c67916fedc" path="/var/lib/kubelet/pods/7f5af3c4-6fdf-4e8d-931b-b4c67916fedc/volumes"
Sep 17 14:12:21.800652 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:21.800618    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:22.368616 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:22.368565    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:23.147696 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:23.147669    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:24.063083 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:24.063052    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:24.902041 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:24.901997    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:30.613335 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:30.613301    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:40.759190 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:40.759154    2560 generic.go:356] "Generic (PLEG): container finished" podID="9276b8fd-fd62-46b3-bdd5-612a7a39b59b" containerID="d05cc22ffddc4253c9e6cd0c3cc987867a3a4d2be00ee503f627cde7c8fbc0f4" exitCode=0
Sep 17 14:12:40.759637 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:40.759229    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672" event={"ID":"9276b8fd-fd62-46b3-bdd5-612a7a39b59b","Type":"ContainerDied","Data":"d05cc22ffddc4253c9e6cd0c3cc987867a3a4d2be00ee503f627cde7c8fbc0f4"}
Sep 17 14:12:40.759685 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:40.759643    2560 scope.go:117] "RemoveContainer" containerID="d05cc22ffddc4253c9e6cd0c3cc987867a3a4d2be00ee503f627cde7c8fbc0f4"
Sep 17 14:12:40.760722 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:40.760692    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:12:40.760837 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:40.760772    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:12:40.760837 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:40.760818    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:12:41.064726 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:41.064698    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dd410a86-gwlb8"]
Sep 17 14:12:41.065229 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:41.065210    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f5af3c4-6fdf-4e8d-931b-b4c67916fedc" containerName="istio-proxy"
Sep 17 14:12:41.065302 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:41.065231    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f5af3c4-6fdf-4e8d-931b-b4c67916fedc" containerName="istio-proxy"
Sep 17 14:12:41.065302 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:41.065249    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f5af3c4-6fdf-4e8d-931b-b4c67916fedc" containerName="istio-proxy"
Sep 17 14:12:41.065302 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:41.065254    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f5af3c4-6fdf-4e8d-931b-b4c67916fedc" containerName="istio-proxy"
Sep 17 14:12:41.065398 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:41.065339    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f5af3c4-6fdf-4e8d-931b-b4c67916fedc" containerName="istio-proxy"
Sep 17 14:12:41.065398 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:41.065349    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f5af3c4-6fdf-4e8d-931b-b4c67916fedc" containerName="istio-proxy"
Sep 17 14:12:41.067341 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:41.067326    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dd410a86-gwlb8"
Sep 17 14:12:41.069755 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:41.069728    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rp9ww\""
Sep 17 14:12:41.069876 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:41.069753    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 14:12:41.077022 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:41.076993    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dd410a86-gwlb8"]
Sep 17 14:12:41.098328 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:41.098304    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ac72507-7ac9-48ec-9758-af0b6ea41854-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-dd410a86-gwlb8\" (UID: \"4ac72507-7ac9-48ec-9758-af0b6ea41854\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dd410a86-gwlb8"
Sep 17 14:12:41.199191 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:41.199157    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ac72507-7ac9-48ec-9758-af0b6ea41854-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-dd410a86-gwlb8\" (UID: \"4ac72507-7ac9-48ec-9758-af0b6ea41854\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dd410a86-gwlb8"
Sep 17 14:12:41.199757 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:41.199734    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ac72507-7ac9-48ec-9758-af0b6ea41854-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-dd410a86-gwlb8\" (UID: \"4ac72507-7ac9-48ec-9758-af0b6ea41854\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dd410a86-gwlb8"
Sep 17 14:12:41.380397 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:41.380291    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dd410a86-gwlb8"
Sep 17 14:12:41.545803 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:41.545779    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:41.654539 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:41.654517    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dd410a86-gwlb8"]
Sep 17 14:12:41.773385 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:41.773341    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672" event={"ID":"9276b8fd-fd62-46b3-bdd5-612a7a39b59b","Type":"ContainerStarted","Data":"7a0ea09c8f32ac9d05fbb477d4a2124c6ccd375abd607d7743e0ca81b749be04"}
Sep 17 14:12:41.776014 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:41.775983    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dd410a86-gwlb8" event={"ID":"4ac72507-7ac9-48ec-9758-af0b6ea41854","Type":"ContainerStarted","Data":"009baf40e5e617ba3a00dbcb3fd1ace2649d3d9901a9887db9d4db66629d7d26"}
Sep 17 14:12:42.564163 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:42.564115    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:42.569918 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:42.569894    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672"
Sep 17 14:12:42.570175 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:42.570154    2560 patch_prober.go:28] interesting pod/e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 17 14:12:42.570347 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:42.570322    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672" podUID="9276b8fd-fd62-46b3-bdd5-612a7a39b59b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 17 14:12:42.786666 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:42.786633    2560 generic.go:356] "Generic (PLEG): container finished" podID="4ac72507-7ac9-48ec-9758-af0b6ea41854" containerID="383110f642adc0f506333b66d5c00e2faada7b0f2e36848ee38cd3004cad8bcf" exitCode=0
Sep 17 14:12:42.787066 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:42.786724    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dd410a86-gwlb8" event={"ID":"4ac72507-7ac9-48ec-9758-af0b6ea41854","Type":"ContainerDied","Data":"383110f642adc0f506333b66d5c00e2faada7b0f2e36848ee38cd3004cad8bcf"}
Sep 17 14:12:43.557168 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:43.557126    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:43.577618 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:43.577583    2560 patch_prober.go:28] interesting pod/e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 17 14:12:43.577799 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:43.577642    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672" podUID="9276b8fd-fd62-46b3-bdd5-612a7a39b59b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 17 14:12:43.938403 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:43.938380    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dd410a86-gwlb8"
Sep 17 14:12:44.033255 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:44.033213    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ac72507-7ac9-48ec-9758-af0b6ea41854-maas-api-service-ca\") pod \"4ac72507-7ac9-48ec-9758-af0b6ea41854\" (UID: \"4ac72507-7ac9-48ec-9758-af0b6ea41854\") "
Sep 17 14:12:44.033557 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:44.033536    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ac72507-7ac9-48ec-9758-af0b6ea41854-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4ac72507-7ac9-48ec-9758-af0b6ea41854" (UID: "4ac72507-7ac9-48ec-9758-af0b6ea41854"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:12:44.033617 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:44.033605    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ac72507-7ac9-48ec-9758-af0b6ea41854-maas-api-service-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:12:44.404264 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:44.404227    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:44.569524 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:44.569490    2560 patch_prober.go:28] interesting pod/e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 17 14:12:44.569786 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:44.569759    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672" podUID="9276b8fd-fd62-46b3-bdd5-612a7a39b59b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 17 14:12:44.799990 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:44.799959    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dd410a86-gwlb8" event={"ID":"4ac72507-7ac9-48ec-9758-af0b6ea41854","Type":"ContainerDied","Data":"009baf40e5e617ba3a00dbcb3fd1ace2649d3d9901a9887db9d4db66629d7d26"}
Sep 17 14:12:44.800156 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:44.800031    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="009baf40e5e617ba3a00dbcb3fd1ace2649d3d9901a9887db9d4db66629d7d26"
Sep 17 14:12:44.800156 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:44.799969    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dd410a86-gwlb8"
Sep 17 14:12:45.456823 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:45.456788    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:45.569606 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:45.569570    2560 patch_prober.go:28] interesting pod/e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 17 14:12:45.569872 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:45.569840    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672" podUID="9276b8fd-fd62-46b3-bdd5-612a7a39b59b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 17 14:12:45.593160 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:45.593131    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672"
Sep 17 14:12:46.402378 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:46.402344    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:46.569191 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:46.569154    2560 patch_prober.go:28] interesting pod/e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 17 14:12:46.569763 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:46.569730    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672" podUID="9276b8fd-fd62-46b3-bdd5-612a7a39b59b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 17 14:12:46.624919 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:46.624890    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:47.103946 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:47.103905    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:47.204065 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:47.204032    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:47.305603 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:47.305564    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:47.377838 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:47.377759    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dd410a86-gwlb8"]
Sep 17 14:12:47.379306 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:47.379269    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:47.380523 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:47.380503    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dd410a86-gwlb8"]
Sep 17 14:12:47.520970 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:47.520936    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672"]
Sep 17 14:12:47.521188 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:47.521149    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672" podUID="9276b8fd-fd62-46b3-bdd5-612a7a39b59b" containerName="istio-proxy" containerID="cri-o://7a0ea09c8f32ac9d05fbb477d4a2124c6ccd375abd607d7743e0ca81b749be04" gracePeriod=30
Sep 17 14:12:47.532850 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:47.532814    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:47.573327 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:47.573294    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ac72507-7ac9-48ec-9758-af0b6ea41854" path="/var/lib/kubelet/pods/4ac72507-7ac9-48ec-9758-af0b6ea41854/volumes"
Sep 17 14:12:47.632524 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:47.632396    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:47.977439 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:47.977355    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:48.303924 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:48.303893    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww"]
Sep 17 14:12:48.304374 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:48.304360    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ac72507-7ac9-48ec-9758-af0b6ea41854" containerName="revoke-keys"
Sep 17 14:12:48.304427 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:48.304375    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ac72507-7ac9-48ec-9758-af0b6ea41854" containerName="revoke-keys"
Sep 17 14:12:48.304481 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:48.304474    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ac72507-7ac9-48ec-9758-af0b6ea41854" containerName="revoke-keys"
Sep 17 14:12:48.306608 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:48.306591    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww"
Sep 17 14:12:48.308969 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:48.308953    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-1c07ea69-openshift-default-dockercfg-sb82j\""
Sep 17 14:12:48.324628 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:48.324605    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww"]
Sep 17 14:12:48.328400 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:48.328375    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:48.377669 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:48.377641    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f738b33-f6f3-43a6-9497-ba53d4595120-workload-certs\") pod \"e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww\" (UID: \"9f738b33-f6f3-43a6-9497-ba53d4595120\") " pod="openshift-ingress/e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww"
Sep 17 14:12:48.377794 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:48.377708    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j6dw\" (UniqueName: \"kubernetes.io/projected/9f738b33-f6f3-43a6-9497-ba53d4595120-kube-api-access-8j6dw\") pod \"e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww\" (UID: \"9f738b33-f6f3-43a6-9497-ba53d4595120\") " pod="openshift-ingress/e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww"
Sep 17 14:12:48.377794 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:48.377742    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f738b33-f6f3-43a6-9497-ba53d4595120-workload-socket\") pod \"e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww\" (UID: \"9f738b33-f6f3-43a6-9497-ba53d4595120\") " pod="openshift-ingress/e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww"
Sep 17 14:12:48.377794 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:48.377769    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f738b33-f6f3-43a6-9497-ba53d4595120-istio-data\") pod \"e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww\" (UID: \"9f738b33-f6f3-43a6-9497-ba53d4595120\") " pod="openshift-ingress/e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww"
Sep 17 14:12:48.377926 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:48.377797    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f738b33-f6f3-43a6-9497-ba53d4595120-istio-token\") pod \"e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww\" (UID: \"9f738b33-f6f3-43a6-9497-ba53d4595120\") " pod="openshift-ingress/e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww"
Sep 17 14:12:48.377926 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:48.377869    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f738b33-f6f3-43a6-9497-ba53d4595120-credential-socket\") pod \"e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww\" (UID: \"9f738b33-f6f3-43a6-9497-ba53d4595120\") " pod="openshift-ingress/e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww"
Sep 17 14:12:48.377926 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:48.377891    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f738b33-f6f3-43a6-9497-ba53d4595120-istio-podinfo\") pod \"e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww\" (UID: \"9f738b33-f6f3-43a6-9497-ba53d4595120\") " pod="openshift-ingress/e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww"
Sep 17 14:12:48.377926 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:48.377905    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f738b33-f6f3-43a6-9497-ba53d4595120-istiod-ca-cert\") pod \"e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww\" (UID: \"9f738b33-f6f3-43a6-9497-ba53d4595120\") " pod="openshift-ingress/e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww"
Sep 17 14:12:48.378071 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:48.377951    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f738b33-f6f3-43a6-9497-ba53d4595120-istio-envoy\") pod \"e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww\" (UID: \"9f738b33-f6f3-43a6-9497-ba53d4595120\") " pod="openshift-ingress/e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww"
Sep 17 14:12:48.479084 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:48.479056    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f738b33-f6f3-43a6-9497-ba53d4595120-istio-podinfo\") pod \"e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww\" (UID: \"9f738b33-f6f3-43a6-9497-ba53d4595120\") " pod="openshift-ingress/e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww"
Sep 17 14:12:48.479290 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:48.479092    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f738b33-f6f3-43a6-9497-ba53d4595120-istiod-ca-cert\") pod \"e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww\" (UID: \"9f738b33-f6f3-43a6-9497-ba53d4595120\") " pod="openshift-ingress/e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww"
Sep 17 14:12:48.479290 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:48.479144    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f738b33-f6f3-43a6-9497-ba53d4595120-istio-envoy\") pod \"e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww\" (UID: \"9f738b33-f6f3-43a6-9497-ba53d4595120\") " pod="openshift-ingress/e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww"
Sep 17 14:12:48.479290 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:48.479179    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f738b33-f6f3-43a6-9497-ba53d4595120-workload-certs\") pod \"e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww\" (UID: \"9f738b33-f6f3-43a6-9497-ba53d4595120\") " pod="openshift-ingress/e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww"
Sep 17 14:12:48.479290 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:48.479258    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j6dw\" (UniqueName: \"kubernetes.io/projected/9f738b33-f6f3-43a6-9497-ba53d4595120-kube-api-access-8j6dw\") pod \"e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww\" (UID: \"9f738b33-f6f3-43a6-9497-ba53d4595120\") " pod="openshift-ingress/e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww"
Sep 17 14:12:48.479290 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:48.479288    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f738b33-f6f3-43a6-9497-ba53d4595120-workload-socket\") pod \"e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww\" (UID: \"9f738b33-f6f3-43a6-9497-ba53d4595120\") " pod="openshift-ingress/e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww"
Sep 17 14:12:48.479589 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:48.479343    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f738b33-f6f3-43a6-9497-ba53d4595120-istio-data\") pod \"e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww\" (UID: \"9f738b33-f6f3-43a6-9497-ba53d4595120\") " pod="openshift-ingress/e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww"
Sep 17 14:12:48.479589 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:48.479397    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f738b33-f6f3-43a6-9497-ba53d4595120-istio-token\") pod \"e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww\" (UID: \"9f738b33-f6f3-43a6-9497-ba53d4595120\") " pod="openshift-ingress/e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww"
Sep 17 14:12:48.479589 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:48.479454    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f738b33-f6f3-43a6-9497-ba53d4595120-credential-socket\") pod \"e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww\" (UID: \"9f738b33-f6f3-43a6-9497-ba53d4595120\") " pod="openshift-ingress/e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww"
Sep 17 14:12:48.479745 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:48.479661    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f738b33-f6f3-43a6-9497-ba53d4595120-workload-certs\") pod \"e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww\" (UID: \"9f738b33-f6f3-43a6-9497-ba53d4595120\") " pod="openshift-ingress/e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww"
Sep 17 14:12:48.479876 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:48.479825    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f738b33-f6f3-43a6-9497-ba53d4595120-workload-socket\") pod \"e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww\" (UID: \"9f738b33-f6f3-43a6-9497-ba53d4595120\") " pod="openshift-ingress/e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww"
Sep 17 14:12:48.479876 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:48.479848    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f738b33-f6f3-43a6-9497-ba53d4595120-credential-socket\") pod \"e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww\" (UID: \"9f738b33-f6f3-43a6-9497-ba53d4595120\") " pod="openshift-ingress/e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww"
Sep 17 14:12:48.480262 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:48.480237    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f738b33-f6f3-43a6-9497-ba53d4595120-istiod-ca-cert\") pod \"e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww\" (UID: \"9f738b33-f6f3-43a6-9497-ba53d4595120\") " pod="openshift-ingress/e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww"
Sep 17 14:12:48.480335 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:48.480278    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f738b33-f6f3-43a6-9497-ba53d4595120-istio-data\") pod \"e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww\" (UID: \"9f738b33-f6f3-43a6-9497-ba53d4595120\") " pod="openshift-ingress/e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww"
Sep 17 14:12:48.482064 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:48.482041    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f738b33-f6f3-43a6-9497-ba53d4595120-istio-envoy\") pod \"e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww\" (UID: \"9f738b33-f6f3-43a6-9497-ba53d4595120\") " pod="openshift-ingress/e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww"
Sep 17 14:12:48.482221 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:48.482207    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f738b33-f6f3-43a6-9497-ba53d4595120-istio-podinfo\") pod \"e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww\" (UID: \"9f738b33-f6f3-43a6-9497-ba53d4595120\") " pod="openshift-ingress/e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww"
Sep 17 14:12:48.487964 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:48.487943    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f738b33-f6f3-43a6-9497-ba53d4595120-istio-token\") pod \"e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww\" (UID: \"9f738b33-f6f3-43a6-9497-ba53d4595120\") " pod="openshift-ingress/e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww"
Sep 17 14:12:48.488448 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:48.488425    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j6dw\" (UniqueName: \"kubernetes.io/projected/9f738b33-f6f3-43a6-9497-ba53d4595120-kube-api-access-8j6dw\") pod \"e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww\" (UID: \"9f738b33-f6f3-43a6-9497-ba53d4595120\") " pod="openshift-ingress/e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww"
Sep 17 14:12:48.617493 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:48.617405    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww"
Sep 17 14:12:48.646446 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:48.646412    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:48.849236 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:48.849192    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:48.898310 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:48.898275    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww"]
Sep 17 14:12:48.899425 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:12:48.899397    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f738b33_f6f3_43a6_9497_ba53d4595120.slice/crio-5e224f46e56c5b4ac58d33260d590df52ae7f8b47c5289d53af657ce00d23826 WatchSource:0}: Error finding container 5e224f46e56c5b4ac58d33260d590df52ae7f8b47c5289d53af657ce00d23826: Status 404 returned error can't find the container with id 5e224f46e56c5b4ac58d33260d590df52ae7f8b47c5289d53af657ce00d23826
Sep 17 14:12:48.901720 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:48.901681    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:12:48.901846 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:48.901763    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:12:48.901846 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:48.901798    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:12:49.591904 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:49.591869    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:49.832709 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:49.832667    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww" event={"ID":"9f738b33-f6f3-43a6-9497-ba53d4595120","Type":"ContainerStarted","Data":"6c12e1ecb3e03ff2c95fe4565006d98cb47a21f903793dd8589276a44477cb6c"}
Sep 17 14:12:49.833073 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:49.832723    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww" event={"ID":"9f738b33-f6f3-43a6-9497-ba53d4595120","Type":"ContainerStarted","Data":"5e224f46e56c5b4ac58d33260d590df52ae7f8b47c5289d53af657ce00d23826"}
Sep 17 14:12:49.858006 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:49.857915    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww" podStartSLOduration=1.857900786 podStartE2EDuration="1.857900786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:12:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:12:49.855451235 +0000 UTC m=+1038.797594880" watchObservedRunningTime="2026-09-17 14:12:49.857900786 +0000 UTC m=+1038.800044432"
Sep 17 14:12:50.617768 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:50.617738    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww"
Sep 17 14:12:50.623003 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:50.622980    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww"
Sep 17 14:12:50.652290 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:50.652262    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:50.839831 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:50.839803    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww"
Sep 17 14:12:50.840858 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:50.840840    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww"
Sep 17 14:12:51.267379 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:51.267343    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:51.376153 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:51.376118    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:51.601891 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:51.601861    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:52.506821 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:52.506783    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:52.782298 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:52.782275    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672"
Sep 17 14:12:52.852693 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:52.852653    2560 generic.go:356] "Generic (PLEG): container finished" podID="9276b8fd-fd62-46b3-bdd5-612a7a39b59b" containerID="7a0ea09c8f32ac9d05fbb477d4a2124c6ccd375abd607d7743e0ca81b749be04" exitCode=0
Sep 17 14:12:52.852823 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:52.852703    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672"
Sep 17 14:12:52.852823 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:52.852734    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672" event={"ID":"9276b8fd-fd62-46b3-bdd5-612a7a39b59b","Type":"ContainerDied","Data":"7a0ea09c8f32ac9d05fbb477d4a2124c6ccd375abd607d7743e0ca81b749be04"}
Sep 17 14:12:52.852823 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:52.852785    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672" event={"ID":"9276b8fd-fd62-46b3-bdd5-612a7a39b59b","Type":"ContainerDied","Data":"868f87a7e81da39960bacdf45f3434288d6ea36581b77f9da29da2d43cb29e1d"}
Sep 17 14:12:52.852823 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:52.852808    2560 scope.go:117] "RemoveContainer" containerID="7a0ea09c8f32ac9d05fbb477d4a2124c6ccd375abd607d7743e0ca81b749be04"
Sep 17 14:12:52.862531 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:52.862516    2560 scope.go:117] "RemoveContainer" containerID="d05cc22ffddc4253c9e6cd0c3cc987867a3a4d2be00ee503f627cde7c8fbc0f4"
Sep 17 14:12:52.871140 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:52.871124    2560 scope.go:117] "RemoveContainer" containerID="7a0ea09c8f32ac9d05fbb477d4a2124c6ccd375abd607d7743e0ca81b749be04"
Sep 17 14:12:52.871404 ip-10-0-129-158 kubenswrapper[2560]: E0917 14:12:52.871389    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a0ea09c8f32ac9d05fbb477d4a2124c6ccd375abd607d7743e0ca81b749be04\": container with ID starting with 7a0ea09c8f32ac9d05fbb477d4a2124c6ccd375abd607d7743e0ca81b749be04 not found: ID does not exist" containerID="7a0ea09c8f32ac9d05fbb477d4a2124c6ccd375abd607d7743e0ca81b749be04"
Sep 17 14:12:52.871470 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:52.871411    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a0ea09c8f32ac9d05fbb477d4a2124c6ccd375abd607d7743e0ca81b749be04"} err="failed to get container status \"7a0ea09c8f32ac9d05fbb477d4a2124c6ccd375abd607d7743e0ca81b749be04\": rpc error: code = NotFound desc = could not find container \"7a0ea09c8f32ac9d05fbb477d4a2124c6ccd375abd607d7743e0ca81b749be04\": container with ID starting with 7a0ea09c8f32ac9d05fbb477d4a2124c6ccd375abd607d7743e0ca81b749be04 not found: ID does not exist"
Sep 17 14:12:52.871470 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:52.871426    2560 scope.go:117] "RemoveContainer" containerID="d05cc22ffddc4253c9e6cd0c3cc987867a3a4d2be00ee503f627cde7c8fbc0f4"
Sep 17 14:12:52.871700 ip-10-0-129-158 kubenswrapper[2560]: E0917 14:12:52.871683    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d05cc22ffddc4253c9e6cd0c3cc987867a3a4d2be00ee503f627cde7c8fbc0f4\": container with ID starting with d05cc22ffddc4253c9e6cd0c3cc987867a3a4d2be00ee503f627cde7c8fbc0f4 not found: ID does not exist" containerID="d05cc22ffddc4253c9e6cd0c3cc987867a3a4d2be00ee503f627cde7c8fbc0f4"
Sep 17 14:12:52.871751 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:52.871705    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d05cc22ffddc4253c9e6cd0c3cc987867a3a4d2be00ee503f627cde7c8fbc0f4"} err="failed to get container status \"d05cc22ffddc4253c9e6cd0c3cc987867a3a4d2be00ee503f627cde7c8fbc0f4\": rpc error: code = NotFound desc = could not find container \"d05cc22ffddc4253c9e6cd0c3cc987867a3a4d2be00ee503f627cde7c8fbc0f4\": container with ID starting with d05cc22ffddc4253c9e6cd0c3cc987867a3a4d2be00ee503f627cde7c8fbc0f4 not found: ID does not exist"
Sep 17 14:12:52.925052 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:52.925027    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9276b8fd-fd62-46b3-bdd5-612a7a39b59b-istio-token\") pod \"9276b8fd-fd62-46b3-bdd5-612a7a39b59b\" (UID: \"9276b8fd-fd62-46b3-bdd5-612a7a39b59b\") "
Sep 17 14:12:52.925149 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:52.925063    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9276b8fd-fd62-46b3-bdd5-612a7a39b59b-workload-socket\") pod \"9276b8fd-fd62-46b3-bdd5-612a7a39b59b\" (UID: \"9276b8fd-fd62-46b3-bdd5-612a7a39b59b\") "
Sep 17 14:12:52.925149 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:52.925101    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9276b8fd-fd62-46b3-bdd5-612a7a39b59b-credential-socket\") pod \"9276b8fd-fd62-46b3-bdd5-612a7a39b59b\" (UID: \"9276b8fd-fd62-46b3-bdd5-612a7a39b59b\") "
Sep 17 14:12:52.925149 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:52.925119    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9276b8fd-fd62-46b3-bdd5-612a7a39b59b-workload-certs\") pod \"9276b8fd-fd62-46b3-bdd5-612a7a39b59b\" (UID: \"9276b8fd-fd62-46b3-bdd5-612a7a39b59b\") "
Sep 17 14:12:52.925265 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:52.925178    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9276b8fd-fd62-46b3-bdd5-612a7a39b59b-istio-podinfo\") pod \"9276b8fd-fd62-46b3-bdd5-612a7a39b59b\" (UID: \"9276b8fd-fd62-46b3-bdd5-612a7a39b59b\") "
Sep 17 14:12:52.925265 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:52.925238    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9276b8fd-fd62-46b3-bdd5-612a7a39b59b-istio-envoy\") pod \"9276b8fd-fd62-46b3-bdd5-612a7a39b59b\" (UID: \"9276b8fd-fd62-46b3-bdd5-612a7a39b59b\") "
Sep 17 14:12:52.925357 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:52.925267    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9276b8fd-fd62-46b3-bdd5-612a7a39b59b-istio-data\") pod \"9276b8fd-fd62-46b3-bdd5-612a7a39b59b\" (UID: \"9276b8fd-fd62-46b3-bdd5-612a7a39b59b\") "
Sep 17 14:12:52.925357 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:52.925294    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7fq8s\" (UniqueName: \"kubernetes.io/projected/9276b8fd-fd62-46b3-bdd5-612a7a39b59b-kube-api-access-7fq8s\") pod \"9276b8fd-fd62-46b3-bdd5-612a7a39b59b\" (UID: \"9276b8fd-fd62-46b3-bdd5-612a7a39b59b\") "
Sep 17 14:12:52.925475 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:52.925350    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9276b8fd-fd62-46b3-bdd5-612a7a39b59b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9276b8fd-fd62-46b3-bdd5-612a7a39b59b" (UID: "9276b8fd-fd62-46b3-bdd5-612a7a39b59b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:12:52.925475 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:52.925362    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9276b8fd-fd62-46b3-bdd5-612a7a39b59b-istiod-ca-cert\") pod \"9276b8fd-fd62-46b3-bdd5-612a7a39b59b\" (UID: \"9276b8fd-fd62-46b3-bdd5-612a7a39b59b\") "
Sep 17 14:12:52.925648 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:52.925453    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9276b8fd-fd62-46b3-bdd5-612a7a39b59b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9276b8fd-fd62-46b3-bdd5-612a7a39b59b" (UID: "9276b8fd-fd62-46b3-bdd5-612a7a39b59b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:12:52.925648 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:52.925550    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9276b8fd-fd62-46b3-bdd5-612a7a39b59b-istio-data" (OuterVolumeSpecName: "istio-data") pod "9276b8fd-fd62-46b3-bdd5-612a7a39b59b" (UID: "9276b8fd-fd62-46b3-bdd5-612a7a39b59b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:12:52.925648 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:52.925591    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9276b8fd-fd62-46b3-bdd5-612a7a39b59b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9276b8fd-fd62-46b3-bdd5-612a7a39b59b" (UID: "9276b8fd-fd62-46b3-bdd5-612a7a39b59b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:12:52.925811 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:52.925665    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9276b8fd-fd62-46b3-bdd5-612a7a39b59b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9276b8fd-fd62-46b3-bdd5-612a7a39b59b" (UID: "9276b8fd-fd62-46b3-bdd5-612a7a39b59b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:12:52.926181 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:52.925930    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9276b8fd-fd62-46b3-bdd5-612a7a39b59b-istiod-ca-cert\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:12:52.926181 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:52.925953    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9276b8fd-fd62-46b3-bdd5-612a7a39b59b-workload-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:12:52.926181 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:52.925968    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9276b8fd-fd62-46b3-bdd5-612a7a39b59b-credential-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:12:52.926181 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:52.925980    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9276b8fd-fd62-46b3-bdd5-612a7a39b59b-workload-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:12:52.926181 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:52.925992    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9276b8fd-fd62-46b3-bdd5-612a7a39b59b-istio-data\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:12:52.927235 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:52.927213    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9276b8fd-fd62-46b3-bdd5-612a7a39b59b-istio-token" (OuterVolumeSpecName: "istio-token") pod "9276b8fd-fd62-46b3-bdd5-612a7a39b59b" (UID: "9276b8fd-fd62-46b3-bdd5-612a7a39b59b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:12:52.927436 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:52.927414    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9276b8fd-fd62-46b3-bdd5-612a7a39b59b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9276b8fd-fd62-46b3-bdd5-612a7a39b59b" (UID: "9276b8fd-fd62-46b3-bdd5-612a7a39b59b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 14:12:52.927604 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:52.927584    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9276b8fd-fd62-46b3-bdd5-612a7a39b59b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9276b8fd-fd62-46b3-bdd5-612a7a39b59b" (UID: "9276b8fd-fd62-46b3-bdd5-612a7a39b59b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:12:52.927667 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:52.927619    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9276b8fd-fd62-46b3-bdd5-612a7a39b59b-kube-api-access-7fq8s" (OuterVolumeSpecName: "kube-api-access-7fq8s") pod "9276b8fd-fd62-46b3-bdd5-612a7a39b59b" (UID: "9276b8fd-fd62-46b3-bdd5-612a7a39b59b"). InnerVolumeSpecName "kube-api-access-7fq8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:12:53.027554 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:53.027451    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9276b8fd-fd62-46b3-bdd5-612a7a39b59b-istio-token\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:12:53.027554 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:53.027508    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9276b8fd-fd62-46b3-bdd5-612a7a39b59b-istio-podinfo\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:12:53.027554 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:53.027517    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9276b8fd-fd62-46b3-bdd5-612a7a39b59b-istio-envoy\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:12:53.027554 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:53.027525    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fq8s\" (UniqueName: \"kubernetes.io/projected/9276b8fd-fd62-46b3-bdd5-612a7a39b59b-kube-api-access-7fq8s\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:12:53.176020 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:53.175993    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672"]
Sep 17 14:12:53.179758 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:53.179736    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-dd410a86-gw-openshift-default-6749c4c4cb-2k672"]
Sep 17 14:12:53.278653 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:53.278577    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:53.571115 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:53.571085    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9276b8fd-fd62-46b3-bdd5-612a7a39b59b" path="/var/lib/kubelet/pods/9276b8fd-fd62-46b3-bdd5-612a7a39b59b/volumes"
Sep 17 14:12:54.200990 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:54.200959    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:54.313682 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:54.313651    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:54.383575 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:54.383540    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:54.717974 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:54.717948    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:54.812254 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:54.812218    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:55.143309 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:55.143266    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:56.078455 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:56.078419    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:56.826361 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:12:56.826324    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:13:13.303206 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:13.303178    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-0458f0/autores-model-0458f0-kserve-6cd946794c-5p2zp"]
Sep 17 14:13:13.303616 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:13.303603    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9276b8fd-fd62-46b3-bdd5-612a7a39b59b" containerName="istio-proxy"
Sep 17 14:13:13.303665 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:13.303618    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9276b8fd-fd62-46b3-bdd5-612a7a39b59b" containerName="istio-proxy"
Sep 17 14:13:13.303701 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:13.303696    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="9276b8fd-fd62-46b3-bdd5-612a7a39b59b" containerName="istio-proxy"
Sep 17 14:13:13.303734 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:13.303704    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="9276b8fd-fd62-46b3-bdd5-612a7a39b59b" containerName="istio-proxy"
Sep 17 14:13:13.306722 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:13.306706    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-0458f0/autores-model-0458f0-kserve-6cd946794c-5p2zp"
Sep 17 14:13:13.309553 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:13.309529    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-0458f0\"/\"kube-root-ca.crt\""
Sep 17 14:13:13.309684 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:13.309564    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-0458f0\"/\"openshift-service-ca.crt\""
Sep 17 14:13:13.309684 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:13.309529    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-0458f0\"/\"default-dockercfg-7wfbm\""
Sep 17 14:13:13.309684 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:13.309533    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-0458f0\"/\"autores-model-0458f0-kserve-self-signed-certs\""
Sep 17 14:13:13.316026 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:13.316005    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-0458f0/autores-model-0458f0-kserve-6cd946794c-5p2zp"]
Sep 17 14:13:13.328286 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:13.328259    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:13:13.424290 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:13.424260    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjwpv\" (UniqueName: \"kubernetes.io/projected/cfc3d9a0-e8cf-44d2-882b-3dc079266498-kube-api-access-gjwpv\") pod \"autores-model-0458f0-kserve-6cd946794c-5p2zp\" (UID: \"cfc3d9a0-e8cf-44d2-882b-3dc079266498\") " pod="ai-tenant-e2e-autores-0458f0/autores-model-0458f0-kserve-6cd946794c-5p2zp"
Sep 17 14:13:13.424487 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:13.424298    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfc3d9a0-e8cf-44d2-882b-3dc079266498-model-cache\") pod \"autores-model-0458f0-kserve-6cd946794c-5p2zp\" (UID: \"cfc3d9a0-e8cf-44d2-882b-3dc079266498\") " pod="ai-tenant-e2e-autores-0458f0/autores-model-0458f0-kserve-6cd946794c-5p2zp"
Sep 17 14:13:13.424487 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:13.424350    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfc3d9a0-e8cf-44d2-882b-3dc079266498-tmp-dir\") pod \"autores-model-0458f0-kserve-6cd946794c-5p2zp\" (UID: \"cfc3d9a0-e8cf-44d2-882b-3dc079266498\") " pod="ai-tenant-e2e-autores-0458f0/autores-model-0458f0-kserve-6cd946794c-5p2zp"
Sep 17 14:13:13.424579 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:13.424485    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfc3d9a0-e8cf-44d2-882b-3dc079266498-dshm\") pod \"autores-model-0458f0-kserve-6cd946794c-5p2zp\" (UID: \"cfc3d9a0-e8cf-44d2-882b-3dc079266498\") " pod="ai-tenant-e2e-autores-0458f0/autores-model-0458f0-kserve-6cd946794c-5p2zp"
Sep 17 14:13:13.424579 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:13.424529    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfc3d9a0-e8cf-44d2-882b-3dc079266498-tls-certs\") pod \"autores-model-0458f0-kserve-6cd946794c-5p2zp\" (UID: \"cfc3d9a0-e8cf-44d2-882b-3dc079266498\") " pod="ai-tenant-e2e-autores-0458f0/autores-model-0458f0-kserve-6cd946794c-5p2zp"
Sep 17 14:13:13.424579 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:13.424574    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfc3d9a0-e8cf-44d2-882b-3dc079266498-home\") pod \"autores-model-0458f0-kserve-6cd946794c-5p2zp\" (UID: \"cfc3d9a0-e8cf-44d2-882b-3dc079266498\") " pod="ai-tenant-e2e-autores-0458f0/autores-model-0458f0-kserve-6cd946794c-5p2zp"
Sep 17 14:13:13.525395 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:13.525360    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfc3d9a0-e8cf-44d2-882b-3dc079266498-dshm\") pod \"autores-model-0458f0-kserve-6cd946794c-5p2zp\" (UID: \"cfc3d9a0-e8cf-44d2-882b-3dc079266498\") " pod="ai-tenant-e2e-autores-0458f0/autores-model-0458f0-kserve-6cd946794c-5p2zp"
Sep 17 14:13:13.525395 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:13.525398    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfc3d9a0-e8cf-44d2-882b-3dc079266498-tls-certs\") pod \"autores-model-0458f0-kserve-6cd946794c-5p2zp\" (UID: \"cfc3d9a0-e8cf-44d2-882b-3dc079266498\") " pod="ai-tenant-e2e-autores-0458f0/autores-model-0458f0-kserve-6cd946794c-5p2zp"
Sep 17 14:13:13.525654 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:13.525445    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfc3d9a0-e8cf-44d2-882b-3dc079266498-home\") pod \"autores-model-0458f0-kserve-6cd946794c-5p2zp\" (UID: \"cfc3d9a0-e8cf-44d2-882b-3dc079266498\") " pod="ai-tenant-e2e-autores-0458f0/autores-model-0458f0-kserve-6cd946794c-5p2zp"
Sep 17 14:13:13.525654 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:13.525557    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjwpv\" (UniqueName: \"kubernetes.io/projected/cfc3d9a0-e8cf-44d2-882b-3dc079266498-kube-api-access-gjwpv\") pod \"autores-model-0458f0-kserve-6cd946794c-5p2zp\" (UID: \"cfc3d9a0-e8cf-44d2-882b-3dc079266498\") " pod="ai-tenant-e2e-autores-0458f0/autores-model-0458f0-kserve-6cd946794c-5p2zp"
Sep 17 14:13:13.525654 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:13.525588    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfc3d9a0-e8cf-44d2-882b-3dc079266498-model-cache\") pod \"autores-model-0458f0-kserve-6cd946794c-5p2zp\" (UID: \"cfc3d9a0-e8cf-44d2-882b-3dc079266498\") " pod="ai-tenant-e2e-autores-0458f0/autores-model-0458f0-kserve-6cd946794c-5p2zp"
Sep 17 14:13:13.525813 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:13.525670    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfc3d9a0-e8cf-44d2-882b-3dc079266498-tmp-dir\") pod \"autores-model-0458f0-kserve-6cd946794c-5p2zp\" (UID: \"cfc3d9a0-e8cf-44d2-882b-3dc079266498\") " pod="ai-tenant-e2e-autores-0458f0/autores-model-0458f0-kserve-6cd946794c-5p2zp"
Sep 17 14:13:13.525926 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:13.525903    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfc3d9a0-e8cf-44d2-882b-3dc079266498-home\") pod \"autores-model-0458f0-kserve-6cd946794c-5p2zp\" (UID: \"cfc3d9a0-e8cf-44d2-882b-3dc079266498\") " pod="ai-tenant-e2e-autores-0458f0/autores-model-0458f0-kserve-6cd946794c-5p2zp"
Sep 17 14:13:13.526121 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:13.526099    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfc3d9a0-e8cf-44d2-882b-3dc079266498-model-cache\") pod \"autores-model-0458f0-kserve-6cd946794c-5p2zp\" (UID: \"cfc3d9a0-e8cf-44d2-882b-3dc079266498\") " pod="ai-tenant-e2e-autores-0458f0/autores-model-0458f0-kserve-6cd946794c-5p2zp"
Sep 17 14:13:13.526217 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:13.526207    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfc3d9a0-e8cf-44d2-882b-3dc079266498-tmp-dir\") pod \"autores-model-0458f0-kserve-6cd946794c-5p2zp\" (UID: \"cfc3d9a0-e8cf-44d2-882b-3dc079266498\") " pod="ai-tenant-e2e-autores-0458f0/autores-model-0458f0-kserve-6cd946794c-5p2zp"
Sep 17 14:13:13.527874 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:13.527850    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfc3d9a0-e8cf-44d2-882b-3dc079266498-dshm\") pod \"autores-model-0458f0-kserve-6cd946794c-5p2zp\" (UID: \"cfc3d9a0-e8cf-44d2-882b-3dc079266498\") " pod="ai-tenant-e2e-autores-0458f0/autores-model-0458f0-kserve-6cd946794c-5p2zp"
Sep 17 14:13:13.528259 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:13.528238    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfc3d9a0-e8cf-44d2-882b-3dc079266498-tls-certs\") pod \"autores-model-0458f0-kserve-6cd946794c-5p2zp\" (UID: \"cfc3d9a0-e8cf-44d2-882b-3dc079266498\") " pod="ai-tenant-e2e-autores-0458f0/autores-model-0458f0-kserve-6cd946794c-5p2zp"
Sep 17 14:13:13.533866 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:13.533844    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjwpv\" (UniqueName: \"kubernetes.io/projected/cfc3d9a0-e8cf-44d2-882b-3dc079266498-kube-api-access-gjwpv\") pod \"autores-model-0458f0-kserve-6cd946794c-5p2zp\" (UID: \"cfc3d9a0-e8cf-44d2-882b-3dc079266498\") " pod="ai-tenant-e2e-autores-0458f0/autores-model-0458f0-kserve-6cd946794c-5p2zp"
Sep 17 14:13:13.617429 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:13.617339    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-0458f0/autores-model-0458f0-kserve-6cd946794c-5p2zp"
Sep 17 14:13:13.899860 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:13.899831    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-0458f0/autores-model-0458f0-kserve-6cd946794c-5p2zp"]
Sep 17 14:13:13.902109 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:13:13.902086    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcfc3d9a0_e8cf_44d2_882b_3dc079266498.slice/crio-c7acef45cc413cb23140305c81fbd5ba23e603147b365a3bc7f9ed51f0323090 WatchSource:0}: Error finding container c7acef45cc413cb23140305c81fbd5ba23e603147b365a3bc7f9ed51f0323090: Status 404 returned error can't find the container with id c7acef45cc413cb23140305c81fbd5ba23e603147b365a3bc7f9ed51f0323090
Sep 17 14:13:14.009454 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:14.009422    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-0458f0/autores-model-0458f0-kserve-6cd946794c-5p2zp" event={"ID":"cfc3d9a0-e8cf-44d2-882b-3dc079266498","Type":"ContainerStarted","Data":"9ad75b0ce23e7581dab9bed8731b23001cd35dfa86c62f29587bfc56dbb4772f"}
Sep 17 14:13:14.009454 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:14.009477    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-0458f0/autores-model-0458f0-kserve-6cd946794c-5p2zp" event={"ID":"cfc3d9a0-e8cf-44d2-882b-3dc079266498","Type":"ContainerStarted","Data":"c7acef45cc413cb23140305c81fbd5ba23e603147b365a3bc7f9ed51f0323090"}
Sep 17 14:13:14.028805 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:14.028752    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-0458f0/autores-model-0458f0-kserve-6cd946794c-5p2zp" podStartSLOduration=1.02873419 podStartE2EDuration="1.02873419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:13:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:13:14.026971087 +0000 UTC m=+1062.969114734" watchObservedRunningTime="2026-09-17 14:13:14.02873419 +0000 UTC m=+1062.970877838"
Sep 17 14:13:16.031847 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:16.031815    2560 generic.go:356] "Generic (PLEG): container finished" podID="9f738b33-f6f3-43a6-9497-ba53d4595120" containerID="6c12e1ecb3e03ff2c95fe4565006d98cb47a21f903793dd8589276a44477cb6c" exitCode=0
Sep 17 14:13:16.032284 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:16.031884    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww" event={"ID":"9f738b33-f6f3-43a6-9497-ba53d4595120","Type":"ContainerDied","Data":"6c12e1ecb3e03ff2c95fe4565006d98cb47a21f903793dd8589276a44477cb6c"}
Sep 17 14:13:16.032337 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:16.032321    2560 scope.go:117] "RemoveContainer" containerID="6c12e1ecb3e03ff2c95fe4565006d98cb47a21f903793dd8589276a44477cb6c"
Sep 17 14:13:16.033511 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:16.033483    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:13:16.033617 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:16.033561    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:13:16.033617 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:16.033606    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:13:16.261020 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:16.260926    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:13:16.993436 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:16.993403    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c07ea69-9x2bg"]
Sep 17 14:13:16.993911 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:16.993896    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9276b8fd-fd62-46b3-bdd5-612a7a39b59b" containerName="istio-proxy"
Sep 17 14:13:16.993968 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:16.993914    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9276b8fd-fd62-46b3-bdd5-612a7a39b59b" containerName="istio-proxy"
Sep 17 14:13:16.997109 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:16.997094    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c07ea69-9x2bg"
Sep 17 14:13:16.999421 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:16.999398    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rp9ww\""
Sep 17 14:13:16.999564 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:16.999477    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 14:13:17.005602 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:17.005062    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c07ea69-9x2bg"]
Sep 17 14:13:17.039270 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:17.039243    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww" event={"ID":"9f738b33-f6f3-43a6-9497-ba53d4595120","Type":"ContainerStarted","Data":"ac73c2970c7f6495bc07ca7dcd2c0c18a193baade3c46bd2da6eac287ccc7913"}
Sep 17 14:13:17.071147 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:17.071111    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fa14982-bec9-41c3-a986-6e532d71b8da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1c07ea69-9x2bg\" (UID: \"0fa14982-bec9-41c3-a986-6e532d71b8da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c07ea69-9x2bg"
Sep 17 14:13:17.179483 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:17.176293    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fa14982-bec9-41c3-a986-6e532d71b8da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1c07ea69-9x2bg\" (UID: \"0fa14982-bec9-41c3-a986-6e532d71b8da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c07ea69-9x2bg"
Sep 17 14:13:17.179483 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:17.178226    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fa14982-bec9-41c3-a986-6e532d71b8da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1c07ea69-9x2bg\" (UID: \"0fa14982-bec9-41c3-a986-6e532d71b8da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c07ea69-9x2bg"
Sep 17 14:13:17.309357 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:17.309325    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c07ea69-9x2bg"
Sep 17 14:13:17.338937 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:17.338909    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:13:17.609050 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:17.609025    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c07ea69-9x2bg"]
Sep 17 14:13:17.610126 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:13:17.610101    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fa14982_bec9_41c3_a986_6e532d71b8da.slice/crio-51db693e757448ef2d1642f5f7776a003b049540764ce1b6270c782a2d4a2ae1 WatchSource:0}: Error finding container 51db693e757448ef2d1642f5f7776a003b049540764ce1b6270c782a2d4a2ae1: Status 404 returned error can't find the container with id 51db693e757448ef2d1642f5f7776a003b049540764ce1b6270c782a2d4a2ae1
Sep 17 14:13:17.618394 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:17.618378    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww"
Sep 17 14:13:17.619409 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:17.619387    2560 patch_prober.go:28] interesting pod/e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Sep 17 14:13:17.619515 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:17.619435    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww" podUID="9f738b33-f6f3-43a6-9497-ba53d4595120" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Sep 17 14:13:18.046678 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:18.046647    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c07ea69-9x2bg" event={"ID":"0fa14982-bec9-41c3-a986-6e532d71b8da","Type":"ContainerStarted","Data":"c881b46a28f4fa8ca75a4a48a1a008737841f80c66328bf9ae03c4723cda492a"}
Sep 17 14:13:18.047019 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:18.046702    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c07ea69-9x2bg" event={"ID":"0fa14982-bec9-41c3-a986-6e532d71b8da","Type":"ContainerStarted","Data":"51db693e757448ef2d1642f5f7776a003b049540764ce1b6270c782a2d4a2ae1"}
Sep 17 14:13:18.064689 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:18.064633    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c07ea69-9x2bg" podStartSLOduration=2.064614178 podStartE2EDuration="2.064614178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:13:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:13:18.063005786 +0000 UTC m=+1067.005149432" watchObservedRunningTime="2026-09-17 14:13:18.064614178 +0000 UTC m=+1067.006757825"
Sep 17 14:13:18.250618 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:18.250531    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:13:18.618157 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:18.618108    2560 patch_prober.go:28] interesting pod/e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Sep 17 14:13:18.618350 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:18.618171    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww" podUID="9f738b33-f6f3-43a6-9497-ba53d4595120" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Sep 17 14:13:19.012783 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:19.012701    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:13:19.054672 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:19.054640    2560 generic.go:356] "Generic (PLEG): container finished" podID="0fa14982-bec9-41c3-a986-6e532d71b8da" containerID="c881b46a28f4fa8ca75a4a48a1a008737841f80c66328bf9ae03c4723cda492a" exitCode=0
Sep 17 14:13:19.055026 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:19.054725    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c07ea69-9x2bg" event={"ID":"0fa14982-bec9-41c3-a986-6e532d71b8da","Type":"ContainerDied","Data":"c881b46a28f4fa8ca75a4a48a1a008737841f80c66328bf9ae03c4723cda492a"}
Sep 17 14:13:19.617832 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:19.617800    2560 patch_prober.go:28] interesting pod/e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Sep 17 14:13:19.618012 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:19.617850    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww" podUID="9f738b33-f6f3-43a6-9497-ba53d4595120" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Sep 17 14:13:19.939199 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:19.939109    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:13:20.238600 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:20.238573    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c07ea69-9x2bg"
Sep 17 14:13:20.316939 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:20.316911    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fa14982-bec9-41c3-a986-6e532d71b8da-maas-api-service-ca\") pod \"0fa14982-bec9-41c3-a986-6e532d71b8da\" (UID: \"0fa14982-bec9-41c3-a986-6e532d71b8da\") "
Sep 17 14:13:20.317299 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:20.317275    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fa14982-bec9-41c3-a986-6e532d71b8da-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0fa14982-bec9-41c3-a986-6e532d71b8da" (UID: "0fa14982-bec9-41c3-a986-6e532d71b8da"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:13:20.425145 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:20.425112    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fa14982-bec9-41c3-a986-6e532d71b8da-maas-api-service-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:13:20.618520 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:20.618488    2560 patch_prober.go:28] interesting pod/e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Sep 17 14:13:20.618691 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:20.618543    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww" podUID="9f738b33-f6f3-43a6-9497-ba53d4595120" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Sep 17 14:13:20.782122 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:20.782087    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:13:20.839803 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:20.839763    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww"
Sep 17 14:13:21.078103 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:21.078070    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c07ea69-9x2bg" event={"ID":"0fa14982-bec9-41c3-a986-6e532d71b8da","Type":"ContainerDied","Data":"51db693e757448ef2d1642f5f7776a003b049540764ce1b6270c782a2d4a2ae1"}
Sep 17 14:13:21.078368 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:21.078350    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51db693e757448ef2d1642f5f7776a003b049540764ce1b6270c782a2d4a2ae1"
Sep 17 14:13:21.078549 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:21.078534    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c07ea69-9x2bg"
Sep 17 14:13:21.617842 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:21.617798    2560 patch_prober.go:28] interesting pod/e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Sep 17 14:13:21.618216 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:21.617854    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww" podUID="9f738b33-f6f3-43a6-9497-ba53d4595120" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Sep 17 14:13:21.738326 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:21.738288    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:13:22.623058 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:22.623025    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww"
Sep 17 14:13:22.624215 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:22.624194    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww"
Sep 17 14:13:22.711773 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:22.711738    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:13:23.248469 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:23.248438    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:13:23.349726 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:23.349695    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:13:23.617550 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:23.617518    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-0458f0/autores-model-0458f0-kserve-6cd946794c-5p2zp"
Sep 17 14:13:23.617736 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:23.617572    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-0458f0/autores-model-0458f0-kserve-6cd946794c-5p2zp"
Sep 17 14:13:23.630052 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:23.630024    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-0458f0/autores-model-0458f0-kserve-6cd946794c-5p2zp"
Sep 17 14:13:23.995094 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:23.994993    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:13:24.107642 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:24.107616    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-0458f0/autores-model-0458f0-kserve-6cd946794c-5p2zp"
Sep 17 14:13:24.191560 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:24.191530    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:13:24.293832 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:24.293745    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:13:24.411072 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:24.411035    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:13:24.452841 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:24.452803    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c07ea69-9x2bg"]
Sep 17 14:13:24.456433 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:24.456407    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c07ea69-9x2bg"]
Sep 17 14:13:24.509714 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:24.509689    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:13:24.881886 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:24.881850    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:13:25.570282 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:25.570247    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0fa14982-bec9-41c3-a986-6e532d71b8da" path="/var/lib/kubelet/pods/0fa14982-bec9-41c3-a986-6e532d71b8da/volumes"
Sep 17 14:13:25.646675 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:25.646644    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww"]
Sep 17 14:13:25.646919 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:25.646877    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww" podUID="9f738b33-f6f3-43a6-9497-ba53d4595120" containerName="istio-proxy" containerID="cri-o://ac73c2970c7f6495bc07ca7dcd2c0c18a193baade3c46bd2da6eac287ccc7913" gracePeriod=30
Sep 17 14:13:25.662825 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:25.662801    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:13:26.043897 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:26.043801    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:13:26.159326 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:26.159287    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:13:26.443526 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:26.443496    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94"]
Sep 17 14:13:26.443954 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:26.443942    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fa14982-bec9-41c3-a986-6e532d71b8da" containerName="revoke-keys"
Sep 17 14:13:26.443996 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:26.443956    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fa14982-bec9-41c3-a986-6e532d71b8da" containerName="revoke-keys"
Sep 17 14:13:26.444053 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:26.444044    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fa14982-bec9-41c3-a986-6e532d71b8da" containerName="revoke-keys"
Sep 17 14:13:26.447372 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:26.447353    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94"
Sep 17 14:13:26.449719 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:26.449697    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-9af6d23e-openshift-default-dockercfg-669tq\""
Sep 17 14:13:26.452006 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:26.451984    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:13:26.467982 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:26.467955    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94"]
Sep 17 14:13:26.588888 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:26.588852    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59b5cd5f-71d1-44a3-acb1-a693b652c4c5-workload-certs\") pod \"e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94\" (UID: \"59b5cd5f-71d1-44a3-acb1-a693b652c4c5\") " pod="openshift-ingress/e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94"
Sep 17 14:13:26.588888 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:26.588890    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59b5cd5f-71d1-44a3-acb1-a693b652c4c5-istio-podinfo\") pod \"e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94\" (UID: \"59b5cd5f-71d1-44a3-acb1-a693b652c4c5\") " pod="openshift-ingress/e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94"
Sep 17 14:13:26.589096 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:26.588999    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59b5cd5f-71d1-44a3-acb1-a693b652c4c5-istio-token\") pod \"e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94\" (UID: \"59b5cd5f-71d1-44a3-acb1-a693b652c4c5\") " pod="openshift-ingress/e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94"
Sep 17 14:13:26.589096 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:26.589040    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b5cd5f-71d1-44a3-acb1-a693b652c4c5-credential-socket\") pod \"e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94\" (UID: \"59b5cd5f-71d1-44a3-acb1-a693b652c4c5\") " pod="openshift-ingress/e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94"
Sep 17 14:13:26.589187 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:26.589111    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59b5cd5f-71d1-44a3-acb1-a693b652c4c5-istiod-ca-cert\") pod \"e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94\" (UID: \"59b5cd5f-71d1-44a3-acb1-a693b652c4c5\") " pod="openshift-ingress/e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94"
Sep 17 14:13:26.589187 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:26.589137    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59b5cd5f-71d1-44a3-acb1-a693b652c4c5-istio-data\") pod \"e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94\" (UID: \"59b5cd5f-71d1-44a3-acb1-a693b652c4c5\") " pod="openshift-ingress/e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94"
Sep 17 14:13:26.589187 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:26.589163    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59b5cd5f-71d1-44a3-acb1-a693b652c4c5-istio-envoy\") pod \"e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94\" (UID: \"59b5cd5f-71d1-44a3-acb1-a693b652c4c5\") " pod="openshift-ingress/e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94"
Sep 17 14:13:26.589307 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:26.589204    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b5cd5f-71d1-44a3-acb1-a693b652c4c5-workload-socket\") pod \"e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94\" (UID: \"59b5cd5f-71d1-44a3-acb1-a693b652c4c5\") " pod="openshift-ingress/e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94"
Sep 17 14:13:26.589307 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:26.589226    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmf6l\" (UniqueName: \"kubernetes.io/projected/59b5cd5f-71d1-44a3-acb1-a693b652c4c5-kube-api-access-kmf6l\") pod \"e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94\" (UID: \"59b5cd5f-71d1-44a3-acb1-a693b652c4c5\") " pod="openshift-ingress/e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94"
Sep 17 14:13:26.690384 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:26.690344    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59b5cd5f-71d1-44a3-acb1-a693b652c4c5-istio-token\") pod \"e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94\" (UID: \"59b5cd5f-71d1-44a3-acb1-a693b652c4c5\") " pod="openshift-ingress/e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94"
Sep 17 14:13:26.690384 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:26.690391    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b5cd5f-71d1-44a3-acb1-a693b652c4c5-credential-socket\") pod \"e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94\" (UID: \"59b5cd5f-71d1-44a3-acb1-a693b652c4c5\") " pod="openshift-ingress/e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94"
Sep 17 14:13:26.690663 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:26.690433    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59b5cd5f-71d1-44a3-acb1-a693b652c4c5-istiod-ca-cert\") pod \"e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94\" (UID: \"59b5cd5f-71d1-44a3-acb1-a693b652c4c5\") " pod="openshift-ingress/e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94"
Sep 17 14:13:26.690663 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:26.690480    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59b5cd5f-71d1-44a3-acb1-a693b652c4c5-istio-data\") pod \"e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94\" (UID: \"59b5cd5f-71d1-44a3-acb1-a693b652c4c5\") " pod="openshift-ingress/e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94"
Sep 17 14:13:26.690663 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:26.690501    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59b5cd5f-71d1-44a3-acb1-a693b652c4c5-istio-envoy\") pod \"e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94\" (UID: \"59b5cd5f-71d1-44a3-acb1-a693b652c4c5\") " pod="openshift-ingress/e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94"
Sep 17 14:13:26.690663 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:26.690625    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b5cd5f-71d1-44a3-acb1-a693b652c4c5-workload-socket\") pod \"e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94\" (UID: \"59b5cd5f-71d1-44a3-acb1-a693b652c4c5\") " pod="openshift-ingress/e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94"
Sep 17 14:13:26.690885 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:26.690675    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmf6l\" (UniqueName: \"kubernetes.io/projected/59b5cd5f-71d1-44a3-acb1-a693b652c4c5-kube-api-access-kmf6l\") pod \"e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94\" (UID: \"59b5cd5f-71d1-44a3-acb1-a693b652c4c5\") " pod="openshift-ingress/e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94"
Sep 17 14:13:26.690885 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:26.690733    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59b5cd5f-71d1-44a3-acb1-a693b652c4c5-workload-certs\") pod \"e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94\" (UID: \"59b5cd5f-71d1-44a3-acb1-a693b652c4c5\") " pod="openshift-ingress/e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94"
Sep 17 14:13:26.690885 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:26.690767    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59b5cd5f-71d1-44a3-acb1-a693b652c4c5-istio-podinfo\") pod \"e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94\" (UID: \"59b5cd5f-71d1-44a3-acb1-a693b652c4c5\") " pod="openshift-ingress/e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94"
Sep 17 14:13:26.691011 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:26.690901    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b5cd5f-71d1-44a3-acb1-a693b652c4c5-credential-socket\") pod \"e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94\" (UID: \"59b5cd5f-71d1-44a3-acb1-a693b652c4c5\") " pod="openshift-ingress/e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94"
Sep 17 14:13:26.691011 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:26.690951    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59b5cd5f-71d1-44a3-acb1-a693b652c4c5-istio-data\") pod \"e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94\" (UID: \"59b5cd5f-71d1-44a3-acb1-a693b652c4c5\") " pod="openshift-ingress/e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94"
Sep 17 14:13:26.691222 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:26.691192    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b5cd5f-71d1-44a3-acb1-a693b652c4c5-workload-socket\") pod \"e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94\" (UID: \"59b5cd5f-71d1-44a3-acb1-a693b652c4c5\") " pod="openshift-ingress/e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94"
Sep 17 14:13:26.691305 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:26.691228    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59b5cd5f-71d1-44a3-acb1-a693b652c4c5-istiod-ca-cert\") pod \"e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94\" (UID: \"59b5cd5f-71d1-44a3-acb1-a693b652c4c5\") " pod="openshift-ingress/e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94"
Sep 17 14:13:26.691305 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:26.691262    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59b5cd5f-71d1-44a3-acb1-a693b652c4c5-workload-certs\") pod \"e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94\" (UID: \"59b5cd5f-71d1-44a3-acb1-a693b652c4c5\") " pod="openshift-ingress/e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94"
Sep 17 14:13:26.693077 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:26.693058    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59b5cd5f-71d1-44a3-acb1-a693b652c4c5-istio-envoy\") pod \"e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94\" (UID: \"59b5cd5f-71d1-44a3-acb1-a693b652c4c5\") " pod="openshift-ingress/e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94"
Sep 17 14:13:26.693203 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:26.693186    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59b5cd5f-71d1-44a3-acb1-a693b652c4c5-istio-podinfo\") pod \"e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94\" (UID: \"59b5cd5f-71d1-44a3-acb1-a693b652c4c5\") " pod="openshift-ingress/e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94"
Sep 17 14:13:26.698677 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:26.698615    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59b5cd5f-71d1-44a3-acb1-a693b652c4c5-istio-token\") pod \"e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94\" (UID: \"59b5cd5f-71d1-44a3-acb1-a693b652c4c5\") " pod="openshift-ingress/e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94"
Sep 17 14:13:26.698756 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:26.698675    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmf6l\" (UniqueName: \"kubernetes.io/projected/59b5cd5f-71d1-44a3-acb1-a693b652c4c5-kube-api-access-kmf6l\") pod \"e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94\" (UID: \"59b5cd5f-71d1-44a3-acb1-a693b652c4c5\") " pod="openshift-ingress/e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94"
Sep 17 14:13:26.758015 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:26.757979    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94"
Sep 17 14:13:26.908214 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:26.908180    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:13:27.037739 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:27.037711    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94"]
Sep 17 14:13:27.038159 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:27.038132    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:13:27.038902 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:13:27.038876    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59b5cd5f_71d1_44a3_acb1_a693b652c4c5.slice/crio-3fbc0f3271d0ee539fa8f8b298e6c1c9973507a45acbc3babb90d7087c09ec8f WatchSource:0}: Error finding container 3fbc0f3271d0ee539fa8f8b298e6c1c9973507a45acbc3babb90d7087c09ec8f: Status 404 returned error can't find the container with id 3fbc0f3271d0ee539fa8f8b298e6c1c9973507a45acbc3babb90d7087c09ec8f
Sep 17 14:13:27.041234 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:27.041193    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:13:27.041322 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:27.041299    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:13:27.041358 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:27.041345    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:13:27.124707 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:27.124671    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94" event={"ID":"59b5cd5f-71d1-44a3-acb1-a693b652c4c5","Type":"ContainerStarted","Data":"8a24814237a5257d743231cff70299f9ef4ec20ba2217e301a9d9bd80b62c834"}
Sep 17 14:13:27.125056 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:27.124733    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94" event={"ID":"59b5cd5f-71d1-44a3-acb1-a693b652c4c5","Type":"ContainerStarted","Data":"3fbc0f3271d0ee539fa8f8b298e6c1c9973507a45acbc3babb90d7087c09ec8f"}
Sep 17 14:13:27.147889 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:27.147840    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94" podStartSLOduration=1.147822801 podStartE2EDuration="1.147822801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:13:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:13:27.144956368 +0000 UTC m=+1076.087100018" watchObservedRunningTime="2026-09-17 14:13:27.147822801 +0000 UTC m=+1076.089966446"
Sep 17 14:13:27.758275 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:27.758244    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94"
Sep 17 14:13:27.861113 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:27.861080    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:13:28.493987 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:28.493957    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:13:28.608882 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:28.608847    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:13:28.763508 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:28.763414    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94"
Sep 17 14:13:28.779575 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:28.779547    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:13:29.137448 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:29.137418    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94"
Sep 17 14:13:29.138365 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:29.138349    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94"
Sep 17 14:13:29.608157 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:29.608122    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:13:30.366919 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:30.366885    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:13:30.469851 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:30.469823    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:13:30.694865 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:30.694838    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:13:30.909400 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:30.909345    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww"
Sep 17 14:13:30.969257 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:30.969226    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:13:31.036932 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:31.036900    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f738b33-f6f3-43a6-9497-ba53d4595120-istio-podinfo\") pod \"9f738b33-f6f3-43a6-9497-ba53d4595120\" (UID: \"9f738b33-f6f3-43a6-9497-ba53d4595120\") "
Sep 17 14:13:31.037074 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:31.036947    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f738b33-f6f3-43a6-9497-ba53d4595120-workload-certs\") pod \"9f738b33-f6f3-43a6-9497-ba53d4595120\" (UID: \"9f738b33-f6f3-43a6-9497-ba53d4595120\") "
Sep 17 14:13:31.037074 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:31.036962    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f738b33-f6f3-43a6-9497-ba53d4595120-istio-envoy\") pod \"9f738b33-f6f3-43a6-9497-ba53d4595120\" (UID: \"9f738b33-f6f3-43a6-9497-ba53d4595120\") "
Sep 17 14:13:31.037074 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:31.036992    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f738b33-f6f3-43a6-9497-ba53d4595120-workload-socket\") pod \"9f738b33-f6f3-43a6-9497-ba53d4595120\" (UID: \"9f738b33-f6f3-43a6-9497-ba53d4595120\") "
Sep 17 14:13:31.037074 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:31.037047    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f738b33-f6f3-43a6-9497-ba53d4595120-istio-data\") pod \"9f738b33-f6f3-43a6-9497-ba53d4595120\" (UID: \"9f738b33-f6f3-43a6-9497-ba53d4595120\") "
Sep 17 14:13:31.037309 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:31.037077    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f738b33-f6f3-43a6-9497-ba53d4595120-istiod-ca-cert\") pod \"9f738b33-f6f3-43a6-9497-ba53d4595120\" (UID: \"9f738b33-f6f3-43a6-9497-ba53d4595120\") "
Sep 17 14:13:31.037309 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:31.037133    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8j6dw\" (UniqueName: \"kubernetes.io/projected/9f738b33-f6f3-43a6-9497-ba53d4595120-kube-api-access-8j6dw\") pod \"9f738b33-f6f3-43a6-9497-ba53d4595120\" (UID: \"9f738b33-f6f3-43a6-9497-ba53d4595120\") "
Sep 17 14:13:31.037309 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:31.037161    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f738b33-f6f3-43a6-9497-ba53d4595120-credential-socket\") pod \"9f738b33-f6f3-43a6-9497-ba53d4595120\" (UID: \"9f738b33-f6f3-43a6-9497-ba53d4595120\") "
Sep 17 14:13:31.037309 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:31.037215    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f738b33-f6f3-43a6-9497-ba53d4595120-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9f738b33-f6f3-43a6-9497-ba53d4595120" (UID: "9f738b33-f6f3-43a6-9497-ba53d4595120"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:13:31.037309 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:31.037245    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f738b33-f6f3-43a6-9497-ba53d4595120-istio-token\") pod \"9f738b33-f6f3-43a6-9497-ba53d4595120\" (UID: \"9f738b33-f6f3-43a6-9497-ba53d4595120\") "
Sep 17 14:13:31.037612 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:31.037404    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f738b33-f6f3-43a6-9497-ba53d4595120-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9f738b33-f6f3-43a6-9497-ba53d4595120" (UID: "9f738b33-f6f3-43a6-9497-ba53d4595120"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:13:31.037612 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:31.037436    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f738b33-f6f3-43a6-9497-ba53d4595120-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9f738b33-f6f3-43a6-9497-ba53d4595120" (UID: "9f738b33-f6f3-43a6-9497-ba53d4595120"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:13:31.037726 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:31.037617    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f738b33-f6f3-43a6-9497-ba53d4595120-istio-data" (OuterVolumeSpecName: "istio-data") pod "9f738b33-f6f3-43a6-9497-ba53d4595120" (UID: "9f738b33-f6f3-43a6-9497-ba53d4595120"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:13:31.037782 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:31.037717    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f738b33-f6f3-43a6-9497-ba53d4595120-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9f738b33-f6f3-43a6-9497-ba53d4595120" (UID: "9f738b33-f6f3-43a6-9497-ba53d4595120"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:13:31.037834 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:31.037776    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f738b33-f6f3-43a6-9497-ba53d4595120-workload-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:13:31.037834 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:31.037796    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f738b33-f6f3-43a6-9497-ba53d4595120-workload-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:13:31.037834 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:31.037809    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f738b33-f6f3-43a6-9497-ba53d4595120-istio-data\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:13:31.037834 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:31.037824    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f738b33-f6f3-43a6-9497-ba53d4595120-istiod-ca-cert\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:13:31.039614 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:31.039590    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f738b33-f6f3-43a6-9497-ba53d4595120-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9f738b33-f6f3-43a6-9497-ba53d4595120" (UID: "9f738b33-f6f3-43a6-9497-ba53d4595120"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 14:13:31.039725 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:31.039608    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f738b33-f6f3-43a6-9497-ba53d4595120-kube-api-access-8j6dw" (OuterVolumeSpecName: "kube-api-access-8j6dw") pod "9f738b33-f6f3-43a6-9497-ba53d4595120" (UID: "9f738b33-f6f3-43a6-9497-ba53d4595120"). InnerVolumeSpecName "kube-api-access-8j6dw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:13:31.039888 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:31.039865    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f738b33-f6f3-43a6-9497-ba53d4595120-istio-token" (OuterVolumeSpecName: "istio-token") pod "9f738b33-f6f3-43a6-9497-ba53d4595120" (UID: "9f738b33-f6f3-43a6-9497-ba53d4595120"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:13:31.039888 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:31.039864    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f738b33-f6f3-43a6-9497-ba53d4595120-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9f738b33-f6f3-43a6-9497-ba53d4595120" (UID: "9f738b33-f6f3-43a6-9497-ba53d4595120"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:13:31.062160 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:31.062131    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:13:31.138706 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:31.138679    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f738b33-f6f3-43a6-9497-ba53d4595120-istio-podinfo\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:13:31.138706 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:31.138702    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f738b33-f6f3-43a6-9497-ba53d4595120-istio-envoy\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:13:31.138869 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:31.138713    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8j6dw\" (UniqueName: \"kubernetes.io/projected/9f738b33-f6f3-43a6-9497-ba53d4595120-kube-api-access-8j6dw\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:13:31.138869 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:31.138722    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f738b33-f6f3-43a6-9497-ba53d4595120-credential-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:13:31.138869 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:31.138730    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f738b33-f6f3-43a6-9497-ba53d4595120-istio-token\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:13:31.151886 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:31.151859    2560 generic.go:356] "Generic (PLEG): container finished" podID="9f738b33-f6f3-43a6-9497-ba53d4595120" containerID="ac73c2970c7f6495bc07ca7dcd2c0c18a193baade3c46bd2da6eac287ccc7913" exitCode=0
Sep 17 14:13:31.152021 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:31.151903    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww"
Sep 17 14:13:31.152021 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:31.151949    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww" event={"ID":"9f738b33-f6f3-43a6-9497-ba53d4595120","Type":"ContainerDied","Data":"ac73c2970c7f6495bc07ca7dcd2c0c18a193baade3c46bd2da6eac287ccc7913"}
Sep 17 14:13:31.152021 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:31.152005    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww" event={"ID":"9f738b33-f6f3-43a6-9497-ba53d4595120","Type":"ContainerDied","Data":"5e224f46e56c5b4ac58d33260d590df52ae7f8b47c5289d53af657ce00d23826"}
Sep 17 14:13:31.152196 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:31.152030    2560 scope.go:117] "RemoveContainer" containerID="ac73c2970c7f6495bc07ca7dcd2c0c18a193baade3c46bd2da6eac287ccc7913"
Sep 17 14:13:31.163051 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:31.163028    2560 scope.go:117] "RemoveContainer" containerID="6c12e1ecb3e03ff2c95fe4565006d98cb47a21f903793dd8589276a44477cb6c"
Sep 17 14:13:31.171640 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:31.171618    2560 scope.go:117] "RemoveContainer" containerID="ac73c2970c7f6495bc07ca7dcd2c0c18a193baade3c46bd2da6eac287ccc7913"
Sep 17 14:13:31.171880 ip-10-0-129-158 kubenswrapper[2560]: E0917 14:13:31.171862    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac73c2970c7f6495bc07ca7dcd2c0c18a193baade3c46bd2da6eac287ccc7913\": container with ID starting with ac73c2970c7f6495bc07ca7dcd2c0c18a193baade3c46bd2da6eac287ccc7913 not found: ID does not exist" containerID="ac73c2970c7f6495bc07ca7dcd2c0c18a193baade3c46bd2da6eac287ccc7913"
Sep 17 14:13:31.171957 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:31.171891    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac73c2970c7f6495bc07ca7dcd2c0c18a193baade3c46bd2da6eac287ccc7913"} err="failed to get container status \"ac73c2970c7f6495bc07ca7dcd2c0c18a193baade3c46bd2da6eac287ccc7913\": rpc error: code = NotFound desc = could not find container \"ac73c2970c7f6495bc07ca7dcd2c0c18a193baade3c46bd2da6eac287ccc7913\": container with ID starting with ac73c2970c7f6495bc07ca7dcd2c0c18a193baade3c46bd2da6eac287ccc7913 not found: ID does not exist"
Sep 17 14:13:31.171957 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:31.171916    2560 scope.go:117] "RemoveContainer" containerID="6c12e1ecb3e03ff2c95fe4565006d98cb47a21f903793dd8589276a44477cb6c"
Sep 17 14:13:31.172167 ip-10-0-129-158 kubenswrapper[2560]: E0917 14:13:31.172150    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c12e1ecb3e03ff2c95fe4565006d98cb47a21f903793dd8589276a44477cb6c\": container with ID starting with 6c12e1ecb3e03ff2c95fe4565006d98cb47a21f903793dd8589276a44477cb6c not found: ID does not exist" containerID="6c12e1ecb3e03ff2c95fe4565006d98cb47a21f903793dd8589276a44477cb6c"
Sep 17 14:13:31.172214 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:31.172178    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c12e1ecb3e03ff2c95fe4565006d98cb47a21f903793dd8589276a44477cb6c"} err="failed to get container status \"6c12e1ecb3e03ff2c95fe4565006d98cb47a21f903793dd8589276a44477cb6c\": rpc error: code = NotFound desc = could not find container \"6c12e1ecb3e03ff2c95fe4565006d98cb47a21f903793dd8589276a44477cb6c\": container with ID starting with 6c12e1ecb3e03ff2c95fe4565006d98cb47a21f903793dd8589276a44477cb6c not found: ID does not exist"
Sep 17 14:13:31.180017 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:31.179989    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww"]
Sep 17 14:13:31.183012 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:31.182988    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-1c07ea69-openshift-default-847d54c5cb-frsww"]
Sep 17 14:13:31.569897 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:31.569863    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f738b33-f6f3-43a6-9497-ba53d4595120" path="/var/lib/kubelet/pods/9f738b33-f6f3-43a6-9497-ba53d4595120/volumes"
Sep 17 14:13:31.878224 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:31.878132    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:13:32.363850 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:32.363819    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:13:33.268845 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:33.268812    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:13:34.076959 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:34.076923    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:13:48.224838 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:48.224802    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0458f0-rst9x"]
Sep 17 14:13:48.225294 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:48.225243    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f738b33-f6f3-43a6-9497-ba53d4595120" containerName="istio-proxy"
Sep 17 14:13:48.225294 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:48.225254    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f738b33-f6f3-43a6-9497-ba53d4595120" containerName="istio-proxy"
Sep 17 14:13:48.225294 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:48.225269    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f738b33-f6f3-43a6-9497-ba53d4595120" containerName="istio-proxy"
Sep 17 14:13:48.225294 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:48.225275    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f738b33-f6f3-43a6-9497-ba53d4595120" containerName="istio-proxy"
Sep 17 14:13:48.225428 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:48.225353    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f738b33-f6f3-43a6-9497-ba53d4595120" containerName="istio-proxy"
Sep 17 14:13:48.225428 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:48.225362    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f738b33-f6f3-43a6-9497-ba53d4595120" containerName="istio-proxy"
Sep 17 14:13:48.229936 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:48.229920    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0458f0-rst9x"
Sep 17 14:13:48.232245 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:48.232213    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 14:13:48.232407 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:48.232315    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rp9ww\""
Sep 17 14:13:48.237630 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:48.237498    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0458f0-rst9x"]
Sep 17 14:13:48.312816 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:48.312780    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f5355fe-05e5-4fc6-954f-2a19869f75cd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-0458f0-rst9x\" (UID: \"8f5355fe-05e5-4fc6-954f-2a19869f75cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0458f0-rst9x"
Sep 17 14:13:48.414056 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:48.414018    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f5355fe-05e5-4fc6-954f-2a19869f75cd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-0458f0-rst9x\" (UID: \"8f5355fe-05e5-4fc6-954f-2a19869f75cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0458f0-rst9x"
Sep 17 14:13:48.414652 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:48.414631    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f5355fe-05e5-4fc6-954f-2a19869f75cd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-0458f0-rst9x\" (UID: \"8f5355fe-05e5-4fc6-954f-2a19869f75cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0458f0-rst9x"
Sep 17 14:13:48.493057 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:48.492976    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:13:48.541832 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:48.541797    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0458f0-rst9x"
Sep 17 14:13:48.840725 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:48.840696    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0458f0-rst9x"]
Sep 17 14:13:48.842064 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:13:48.842041    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f5355fe_05e5_4fc6_954f_2a19869f75cd.slice/crio-0e3975d64e473e9fb63522b912d039c00421c6ea4cac9389ef61073a24aa9823 WatchSource:0}: Error finding container 0e3975d64e473e9fb63522b912d039c00421c6ea4cac9389ef61073a24aa9823: Status 404 returned error can't find the container with id 0e3975d64e473e9fb63522b912d039c00421c6ea4cac9389ef61073a24aa9823
Sep 17 14:13:49.289280 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:49.289251    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0458f0-rst9x" event={"ID":"8f5355fe-05e5-4fc6-954f-2a19869f75cd","Type":"ContainerStarted","Data":"ccac192eab7db273755b327d17ae27f6cd518d4b4564b93ce82aa938797d14a3"}
Sep 17 14:13:49.289620 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:49.289303    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0458f0-rst9x" event={"ID":"8f5355fe-05e5-4fc6-954f-2a19869f75cd","Type":"ContainerStarted","Data":"0e3975d64e473e9fb63522b912d039c00421c6ea4cac9389ef61073a24aa9823"}
Sep 17 14:13:49.307452 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:49.307410    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0458f0-rst9x" podStartSLOduration=1.30739651 podStartE2EDuration="1.30739651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:13:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:13:49.305418785 +0000 UTC m=+1098.247562444" watchObservedRunningTime="2026-09-17 14:13:49.30739651 +0000 UTC m=+1098.249540156"
Sep 17 14:13:49.451775 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:49.451735    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:13:50.246940 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:50.246908    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:13:50.296473 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:50.296440    2560 generic.go:356] "Generic (PLEG): container finished" podID="8f5355fe-05e5-4fc6-954f-2a19869f75cd" containerID="ccac192eab7db273755b327d17ae27f6cd518d4b4564b93ce82aa938797d14a3" exitCode=0
Sep 17 14:13:50.296798 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:50.296508    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0458f0-rst9x" event={"ID":"8f5355fe-05e5-4fc6-954f-2a19869f75cd","Type":"ContainerDied","Data":"ccac192eab7db273755b327d17ae27f6cd518d4b4564b93ce82aa938797d14a3"}
Sep 17 14:13:51.222728 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:51.222685    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:13:51.437081 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:51.437058    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0458f0-rst9x"
Sep 17 14:13:51.547052 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:51.546974    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f5355fe-05e5-4fc6-954f-2a19869f75cd-maas-api-service-ca\") pod \"8f5355fe-05e5-4fc6-954f-2a19869f75cd\" (UID: \"8f5355fe-05e5-4fc6-954f-2a19869f75cd\") "
Sep 17 14:13:51.547311 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:51.547287    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f5355fe-05e5-4fc6-954f-2a19869f75cd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8f5355fe-05e5-4fc6-954f-2a19869f75cd" (UID: "8f5355fe-05e5-4fc6-954f-2a19869f75cd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:13:51.648073 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:51.648033    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f5355fe-05e5-4fc6-954f-2a19869f75cd-maas-api-service-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:13:52.175455 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:52.175416    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:13:52.312223 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:52.312195    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0458f0-rst9x"
Sep 17 14:13:52.312407 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:52.312219    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0458f0-rst9x" event={"ID":"8f5355fe-05e5-4fc6-954f-2a19869f75cd","Type":"ContainerDied","Data":"0e3975d64e473e9fb63522b912d039c00421c6ea4cac9389ef61073a24aa9823"}
Sep 17 14:13:52.312407 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:52.312287    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e3975d64e473e9fb63522b912d039c00421c6ea4cac9389ef61073a24aa9823"
Sep 17 14:13:53.054398 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:53.054368    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:13:53.321114 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:53.321088    2560 generic.go:356] "Generic (PLEG): container finished" podID="59b5cd5f-71d1-44a3-acb1-a693b652c4c5" containerID="8a24814237a5257d743231cff70299f9ef4ec20ba2217e301a9d9bd80b62c834" exitCode=0
Sep 17 14:13:53.321278 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:53.321172    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94" event={"ID":"59b5cd5f-71d1-44a3-acb1-a693b652c4c5","Type":"ContainerDied","Data":"8a24814237a5257d743231cff70299f9ef4ec20ba2217e301a9d9bd80b62c834"}
Sep 17 14:13:53.321595 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:53.321581    2560 scope.go:117] "RemoveContainer" containerID="8a24814237a5257d743231cff70299f9ef4ec20ba2217e301a9d9bd80b62c834"
Sep 17 14:13:53.322803 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:53.322772    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:13:53.322900 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:53.322857    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:13:53.322944 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:53.322910    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:13:54.175691 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:54.175654    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:13:54.328873 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:54.328843    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94" event={"ID":"59b5cd5f-71d1-44a3-acb1-a693b652c4c5","Type":"ContainerStarted","Data":"75ba25abfbdd5161953d0c444c3943fe6752c31a9c6dafef98cfdfde46d56a22"}
Sep 17 14:13:54.760426 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:54.759961    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94"
Sep 17 14:13:54.760426 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:54.760098    2560 patch_prober.go:28] interesting pod/e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Sep 17 14:13:54.760426 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:54.760153    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94" podUID="59b5cd5f-71d1-44a3-acb1-a693b652c4c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Sep 17 14:13:55.252880 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:55.252838    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:13:55.759419 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:55.759373    2560 patch_prober.go:28] interesting pod/e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Sep 17 14:13:55.759642 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:55.759446    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94" podUID="59b5cd5f-71d1-44a3-acb1-a693b652c4c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Sep 17 14:13:56.139749 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:56.139713    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:13:56.758985 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:56.758947    2560 patch_prober.go:28] interesting pod/e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Sep 17 14:13:56.759452 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:56.759013    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94" podUID="59b5cd5f-71d1-44a3-acb1-a693b652c4c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Sep 17 14:13:56.937720 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:56.937679    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:13:57.759319 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:57.759287    2560 patch_prober.go:28] interesting pod/e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Sep 17 14:13:57.759909 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:57.759876    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94" podUID="59b5cd5f-71d1-44a3-acb1-a693b652c4c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Sep 17 14:13:57.872268 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:57.872236    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:13:58.498289 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:58.498251    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9af6d23e-l7tz6"]
Sep 17 14:13:58.498730 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:58.498715    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f5355fe-05e5-4fc6-954f-2a19869f75cd" containerName="revoke-keys"
Sep 17 14:13:58.498805 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:58.498732    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f5355fe-05e5-4fc6-954f-2a19869f75cd" containerName="revoke-keys"
Sep 17 14:13:58.498865 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:58.498812    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f5355fe-05e5-4fc6-954f-2a19869f75cd" containerName="revoke-keys"
Sep 17 14:13:58.502133 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:58.502116    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9af6d23e-l7tz6"
Sep 17 14:13:58.504354 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:58.504325    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 14:13:58.504494 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:58.504391    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rp9ww\""
Sep 17 14:13:58.510004 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:58.509980    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9af6d23e-l7tz6"]
Sep 17 14:13:58.624213 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:58.624177    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db3fa3f8-47d9-44c5-ad4d-c26e218b08e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-9af6d23e-l7tz6\" (UID: \"db3fa3f8-47d9-44c5-ad4d-c26e218b08e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9af6d23e-l7tz6"
Sep 17 14:13:58.725556 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:58.725522    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db3fa3f8-47d9-44c5-ad4d-c26e218b08e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-9af6d23e-l7tz6\" (UID: \"db3fa3f8-47d9-44c5-ad4d-c26e218b08e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9af6d23e-l7tz6"
Sep 17 14:13:58.726191 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:58.726166    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db3fa3f8-47d9-44c5-ad4d-c26e218b08e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-9af6d23e-l7tz6\" (UID: \"db3fa3f8-47d9-44c5-ad4d-c26e218b08e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9af6d23e-l7tz6"
Sep 17 14:13:58.758886 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:58.758817    2560 patch_prober.go:28] interesting pod/e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Sep 17 14:13:58.759075 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:58.759050    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94" podUID="59b5cd5f-71d1-44a3-acb1-a693b652c4c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Sep 17 14:13:58.814945 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:58.814920    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9af6d23e-l7tz6"
Sep 17 14:13:58.849388 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:58.849319    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:13:59.111493 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:59.111472    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9af6d23e-l7tz6"]
Sep 17 14:13:59.113046 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:13:59.113011    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb3fa3f8_47d9_44c5_ad4d_c26e218b08e7.slice/crio-0c35d126e027009597636cc049a23952a402aef4662b56d972a580074848b395 WatchSource:0}: Error finding container 0c35d126e027009597636cc049a23952a402aef4662b56d972a580074848b395: Status 404 returned error can't find the container with id 0c35d126e027009597636cc049a23952a402aef4662b56d972a580074848b395
Sep 17 14:13:59.137627 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:59.137608    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94"
Sep 17 14:13:59.362886 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:59.362810    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9af6d23e-l7tz6" event={"ID":"db3fa3f8-47d9-44c5-ad4d-c26e218b08e7","Type":"ContainerStarted","Data":"0c35d126e027009597636cc049a23952a402aef4662b56d972a580074848b395"}
Sep 17 14:13:59.745085 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:59.744969    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:13:59.762995 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:59.762965    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94"
Sep 17 14:13:59.764090 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:13:59.764071    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94"
Sep 17 14:14:00.370751 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:00.370715    2560 generic.go:356] "Generic (PLEG): container finished" podID="db3fa3f8-47d9-44c5-ad4d-c26e218b08e7" containerID="6cad336385e879ea9f08ec96cea3193f66ed397130f121a7db80342181dd33eb" exitCode=0
Sep 17 14:14:00.371200 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:00.370794    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9af6d23e-l7tz6" event={"ID":"db3fa3f8-47d9-44c5-ad4d-c26e218b08e7","Type":"ContainerDied","Data":"6cad336385e879ea9f08ec96cea3193f66ed397130f121a7db80342181dd33eb"}
Sep 17 14:14:00.530228 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:00.530195    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:01.381293 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:01.381257    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:01.410551 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:01.410519    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-0458f0-6486b5497f-ddqdk"]
Sep 17 14:14:01.410764 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:01.410739    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-0458f0-6486b5497f-ddqdk" podUID="f7e2e06f-693e-49f5-85ae-1f6768663911" containerName="payload-processing" containerID="cri-o://5759e9db32b9d8a691adfda2cf3e99eec28a54eb2ba01dde283d8001729fdc53" gracePeriod=30
Sep 17 14:14:01.481797 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:01.481748    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:01.541835 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:01.541805    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9af6d23e-l7tz6"
Sep 17 14:14:01.563392 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:01.563348    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:01.621208 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:01.621176    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0458f0-rst9x"]
Sep 17 14:14:01.625926 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:01.625897    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0458f0-rst9x"]
Sep 17 14:14:01.643154 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:01.643078    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:01.656147 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:01.656121    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db3fa3f8-47d9-44c5-ad4d-c26e218b08e7-maas-api-service-ca\") pod \"db3fa3f8-47d9-44c5-ad4d-c26e218b08e7\" (UID: \"db3fa3f8-47d9-44c5-ad4d-c26e218b08e7\") "
Sep 17 14:14:01.656537 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:01.656513    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db3fa3f8-47d9-44c5-ad4d-c26e218b08e7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "db3fa3f8-47d9-44c5-ad4d-c26e218b08e7" (UID: "db3fa3f8-47d9-44c5-ad4d-c26e218b08e7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:14:01.684947 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:01.684927    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-0458f0-6486b5497f-ddqdk"
Sep 17 14:14:01.720656 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:01.720625    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:01.757682 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:01.757648    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7smps\" (UniqueName: \"kubernetes.io/projected/f7e2e06f-693e-49f5-85ae-1f6768663911-kube-api-access-7smps\") pod \"f7e2e06f-693e-49f5-85ae-1f6768663911\" (UID: \"f7e2e06f-693e-49f5-85ae-1f6768663911\") "
Sep 17 14:14:01.757923 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:01.757698    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7e2e06f-693e-49f5-85ae-1f6768663911-plugins-config-volume\") pod \"f7e2e06f-693e-49f5-85ae-1f6768663911\" (UID: \"f7e2e06f-693e-49f5-85ae-1f6768663911\") "
Sep 17 14:14:01.759970 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:01.758443    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db3fa3f8-47d9-44c5-ad4d-c26e218b08e7-maas-api-service-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:14:01.759970 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:01.758833    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7e2e06f-693e-49f5-85ae-1f6768663911-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f7e2e06f-693e-49f5-85ae-1f6768663911" (UID: "f7e2e06f-693e-49f5-85ae-1f6768663911"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:14:01.761940 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:01.761899    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7e2e06f-693e-49f5-85ae-1f6768663911-kube-api-access-7smps" (OuterVolumeSpecName: "kube-api-access-7smps") pod "f7e2e06f-693e-49f5-85ae-1f6768663911" (UID: "f7e2e06f-693e-49f5-85ae-1f6768663911"). InnerVolumeSpecName "kube-api-access-7smps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:14:01.859704 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:01.859659    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7smps\" (UniqueName: \"kubernetes.io/projected/f7e2e06f-693e-49f5-85ae-1f6768663911-kube-api-access-7smps\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:14:01.859704 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:01.859697    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7e2e06f-693e-49f5-85ae-1f6768663911-plugins-config-volume\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:14:02.001024 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:02.000920    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:02.384453 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:02.384411    2560 generic.go:356] "Generic (PLEG): container finished" podID="f7e2e06f-693e-49f5-85ae-1f6768663911" containerID="5759e9db32b9d8a691adfda2cf3e99eec28a54eb2ba01dde283d8001729fdc53" exitCode=0
Sep 17 14:14:02.384903 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:02.384475    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-0458f0-6486b5497f-ddqdk"
Sep 17 14:14:02.384903 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:02.384506    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-0458f0-6486b5497f-ddqdk" event={"ID":"f7e2e06f-693e-49f5-85ae-1f6768663911","Type":"ContainerDied","Data":"5759e9db32b9d8a691adfda2cf3e99eec28a54eb2ba01dde283d8001729fdc53"}
Sep 17 14:14:02.384903 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:02.384564    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-0458f0-6486b5497f-ddqdk" event={"ID":"f7e2e06f-693e-49f5-85ae-1f6768663911","Type":"ContainerDied","Data":"557d99b788dc46be8881d3790777d55841f52ff709d06c29c2366bd4d1436b48"}
Sep 17 14:14:02.384903 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:02.384595    2560 scope.go:117] "RemoveContainer" containerID="5759e9db32b9d8a691adfda2cf3e99eec28a54eb2ba01dde283d8001729fdc53"
Sep 17 14:14:02.386216 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:02.386187    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9af6d23e-l7tz6" event={"ID":"db3fa3f8-47d9-44c5-ad4d-c26e218b08e7","Type":"ContainerDied","Data":"0c35d126e027009597636cc049a23952a402aef4662b56d972a580074848b395"}
Sep 17 14:14:02.386310 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:02.386232    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9af6d23e-l7tz6"
Sep 17 14:14:02.386310 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:02.386242    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c35d126e027009597636cc049a23952a402aef4662b56d972a580074848b395"
Sep 17 14:14:02.398672 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:02.398654    2560 scope.go:117] "RemoveContainer" containerID="5759e9db32b9d8a691adfda2cf3e99eec28a54eb2ba01dde283d8001729fdc53"
Sep 17 14:14:02.398974 ip-10-0-129-158 kubenswrapper[2560]: E0917 14:14:02.398956    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5759e9db32b9d8a691adfda2cf3e99eec28a54eb2ba01dde283d8001729fdc53\": container with ID starting with 5759e9db32b9d8a691adfda2cf3e99eec28a54eb2ba01dde283d8001729fdc53 not found: ID does not exist" containerID="5759e9db32b9d8a691adfda2cf3e99eec28a54eb2ba01dde283d8001729fdc53"
Sep 17 14:14:02.399023 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:02.398984    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5759e9db32b9d8a691adfda2cf3e99eec28a54eb2ba01dde283d8001729fdc53"} err="failed to get container status \"5759e9db32b9d8a691adfda2cf3e99eec28a54eb2ba01dde283d8001729fdc53\": rpc error: code = NotFound desc = could not find container \"5759e9db32b9d8a691adfda2cf3e99eec28a54eb2ba01dde283d8001729fdc53\": container with ID starting with 5759e9db32b9d8a691adfda2cf3e99eec28a54eb2ba01dde283d8001729fdc53 not found: ID does not exist"
Sep 17 14:14:02.412015 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:02.411991    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-0458f0-6486b5497f-ddqdk"]
Sep 17 14:14:02.427895 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:02.427874    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-0458f0-6486b5497f-ddqdk"]
Sep 17 14:14:02.677629 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:02.677537    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:03.553342 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:03.553296    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:03.570006 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:03.569972    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f5355fe-05e5-4fc6-954f-2a19869f75cd" path="/var/lib/kubelet/pods/8f5355fe-05e5-4fc6-954f-2a19869f75cd/volumes"
Sep 17 14:14:03.570386 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:03.570372    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7e2e06f-693e-49f5-85ae-1f6768663911" path="/var/lib/kubelet/pods/f7e2e06f-693e-49f5-85ae-1f6768663911/volumes"
Sep 17 14:14:04.341432 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:04.341400    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:05.061878 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:05.061845    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:06.012578 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:06.012536    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:06.993308 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:06.993274    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-0458f0/autores-model-0458f0-kserve-6cd946794c-5p2zp"]
Sep 17 14:14:07.093603 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:07.093571    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:07.187315 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:07.187284    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:07.949719 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:07.949686    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:10.481161 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:10.481129    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:10.605822 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:10.605754    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:10.709908 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:10.709870    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:10.785931 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:10.785853    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9af6d23e-l7tz6"]
Sep 17 14:14:10.788432 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:10.788411    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9af6d23e-l7tz6"]
Sep 17 14:14:10.800830 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:10.800803    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:10.935432 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:10.935400    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94"]
Sep 17 14:14:10.935661 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:10.935634    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94" podUID="59b5cd5f-71d1-44a3-acb1-a693b652c4c5" containerName="istio-proxy" containerID="cri-o://75ba25abfbdd5161953d0c444c3943fe6752c31a9c6dafef98cfdfde46d56a22" gracePeriod=30
Sep 17 14:14:11.175928 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:11.175893    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:11.384536 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:11.384493    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:11.571357 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:11.571314    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db3fa3f8-47d9-44c5-ad4d-c26e218b08e7" path="/var/lib/kubelet/pods/db3fa3f8-47d9-44c5-ad4d-c26e218b08e7/volumes"
Sep 17 14:14:11.814294 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:11.814251    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:11.905262 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:11.905231    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:12.001177 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:12.001143    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:12.604155 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:12.604123    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:13.548669 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:13.548632    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:13.652653 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:13.652616    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:14.568606 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:14.568577    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:14.766223 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:14.766184    2560 patch_prober.go:28] interesting pod/e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 17 14:14:14.766634 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:14.766265    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94" podUID="59b5cd5f-71d1-44a3-acb1-a693b652c4c5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 14:14:15.334337 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:15.334305    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:16.190819 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:16.190799    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94"
Sep 17 14:14:16.301359 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:16.301339    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kmf6l\" (UniqueName: \"kubernetes.io/projected/59b5cd5f-71d1-44a3-acb1-a693b652c4c5-kube-api-access-kmf6l\") pod \"59b5cd5f-71d1-44a3-acb1-a693b652c4c5\" (UID: \"59b5cd5f-71d1-44a3-acb1-a693b652c4c5\") "
Sep 17 14:14:16.301505 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:16.301408    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59b5cd5f-71d1-44a3-acb1-a693b652c4c5-istio-envoy\") pod \"59b5cd5f-71d1-44a3-acb1-a693b652c4c5\" (UID: \"59b5cd5f-71d1-44a3-acb1-a693b652c4c5\") "
Sep 17 14:14:16.301505 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:16.301433    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59b5cd5f-71d1-44a3-acb1-a693b652c4c5-istiod-ca-cert\") pod \"59b5cd5f-71d1-44a3-acb1-a693b652c4c5\" (UID: \"59b5cd5f-71d1-44a3-acb1-a693b652c4c5\") "
Sep 17 14:14:16.301505 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:16.301487    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59b5cd5f-71d1-44a3-acb1-a693b652c4c5-istio-token\") pod \"59b5cd5f-71d1-44a3-acb1-a693b652c4c5\" (UID: \"59b5cd5f-71d1-44a3-acb1-a693b652c4c5\") "
Sep 17 14:14:16.301696 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:16.301529    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b5cd5f-71d1-44a3-acb1-a693b652c4c5-credential-socket\") pod \"59b5cd5f-71d1-44a3-acb1-a693b652c4c5\" (UID: \"59b5cd5f-71d1-44a3-acb1-a693b652c4c5\") "
Sep 17 14:14:16.301696 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:16.301555    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b5cd5f-71d1-44a3-acb1-a693b652c4c5-workload-socket\") pod \"59b5cd5f-71d1-44a3-acb1-a693b652c4c5\" (UID: \"59b5cd5f-71d1-44a3-acb1-a693b652c4c5\") "
Sep 17 14:14:16.301696 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:16.301572    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59b5cd5f-71d1-44a3-acb1-a693b652c4c5-istio-podinfo\") pod \"59b5cd5f-71d1-44a3-acb1-a693b652c4c5\" (UID: \"59b5cd5f-71d1-44a3-acb1-a693b652c4c5\") "
Sep 17 14:14:16.301696 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:16.301600    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59b5cd5f-71d1-44a3-acb1-a693b652c4c5-workload-certs\") pod \"59b5cd5f-71d1-44a3-acb1-a693b652c4c5\" (UID: \"59b5cd5f-71d1-44a3-acb1-a693b652c4c5\") "
Sep 17 14:14:16.301696 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:16.301623    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59b5cd5f-71d1-44a3-acb1-a693b652c4c5-istio-data\") pod \"59b5cd5f-71d1-44a3-acb1-a693b652c4c5\" (UID: \"59b5cd5f-71d1-44a3-acb1-a693b652c4c5\") "
Sep 17 14:14:16.301960 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:16.301818    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59b5cd5f-71d1-44a3-acb1-a693b652c4c5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "59b5cd5f-71d1-44a3-acb1-a693b652c4c5" (UID: "59b5cd5f-71d1-44a3-acb1-a693b652c4c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:14:16.301960 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:16.301858    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59b5cd5f-71d1-44a3-acb1-a693b652c4c5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "59b5cd5f-71d1-44a3-acb1-a693b652c4c5" (UID: "59b5cd5f-71d1-44a3-acb1-a693b652c4c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:14:16.302073 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:16.302031    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59b5cd5f-71d1-44a3-acb1-a693b652c4c5-istio-data" (OuterVolumeSpecName: "istio-data") pod "59b5cd5f-71d1-44a3-acb1-a693b652c4c5" (UID: "59b5cd5f-71d1-44a3-acb1-a693b652c4c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:14:16.302125 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:16.302070    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59b5cd5f-71d1-44a3-acb1-a693b652c4c5-istiod-ca-cert\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:14:16.302125 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:16.302073    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59b5cd5f-71d1-44a3-acb1-a693b652c4c5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "59b5cd5f-71d1-44a3-acb1-a693b652c4c5" (UID: "59b5cd5f-71d1-44a3-acb1-a693b652c4c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:14:16.302125 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:16.302090    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b5cd5f-71d1-44a3-acb1-a693b652c4c5-workload-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:14:16.302282 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:16.302174    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59b5cd5f-71d1-44a3-acb1-a693b652c4c5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "59b5cd5f-71d1-44a3-acb1-a693b652c4c5" (UID: "59b5cd5f-71d1-44a3-acb1-a693b652c4c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:14:16.303654 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:16.303625    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59b5cd5f-71d1-44a3-acb1-a693b652c4c5-kube-api-access-kmf6l" (OuterVolumeSpecName: "kube-api-access-kmf6l") pod "59b5cd5f-71d1-44a3-acb1-a693b652c4c5" (UID: "59b5cd5f-71d1-44a3-acb1-a693b652c4c5"). InnerVolumeSpecName "kube-api-access-kmf6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:14:16.303747 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:16.303678    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59b5cd5f-71d1-44a3-acb1-a693b652c4c5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "59b5cd5f-71d1-44a3-acb1-a693b652c4c5" (UID: "59b5cd5f-71d1-44a3-acb1-a693b652c4c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:14:16.303747 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:16.303691    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59b5cd5f-71d1-44a3-acb1-a693b652c4c5-istio-token" (OuterVolumeSpecName: "istio-token") pod "59b5cd5f-71d1-44a3-acb1-a693b652c4c5" (UID: "59b5cd5f-71d1-44a3-acb1-a693b652c4c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:14:16.303870 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:16.303853    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59b5cd5f-71d1-44a3-acb1-a693b652c4c5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "59b5cd5f-71d1-44a3-acb1-a693b652c4c5" (UID: "59b5cd5f-71d1-44a3-acb1-a693b652c4c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 14:14:16.403312 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:16.403287    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmf6l\" (UniqueName: \"kubernetes.io/projected/59b5cd5f-71d1-44a3-acb1-a693b652c4c5-kube-api-access-kmf6l\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:14:16.403312 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:16.403309    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59b5cd5f-71d1-44a3-acb1-a693b652c4c5-istio-envoy\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:14:16.403473 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:16.403318    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59b5cd5f-71d1-44a3-acb1-a693b652c4c5-istio-token\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:14:16.403473 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:16.403326    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b5cd5f-71d1-44a3-acb1-a693b652c4c5-credential-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:14:16.403473 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:16.403335    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59b5cd5f-71d1-44a3-acb1-a693b652c4c5-istio-podinfo\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:14:16.403473 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:16.403343    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59b5cd5f-71d1-44a3-acb1-a693b652c4c5-workload-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:14:16.403473 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:16.403350    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59b5cd5f-71d1-44a3-acb1-a693b652c4c5-istio-data\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:14:16.478538 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:16.478512    2560 generic.go:356] "Generic (PLEG): container finished" podID="59b5cd5f-71d1-44a3-acb1-a693b652c4c5" containerID="75ba25abfbdd5161953d0c444c3943fe6752c31a9c6dafef98cfdfde46d56a22" exitCode=0
Sep 17 14:14:16.478653 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:16.478557    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94"
Sep 17 14:14:16.478653 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:16.478601    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94" event={"ID":"59b5cd5f-71d1-44a3-acb1-a693b652c4c5","Type":"ContainerDied","Data":"75ba25abfbdd5161953d0c444c3943fe6752c31a9c6dafef98cfdfde46d56a22"}
Sep 17 14:14:16.478653 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:16.478651    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94" event={"ID":"59b5cd5f-71d1-44a3-acb1-a693b652c4c5","Type":"ContainerDied","Data":"3fbc0f3271d0ee539fa8f8b298e6c1c9973507a45acbc3babb90d7087c09ec8f"}
Sep 17 14:14:16.478771 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:16.478673    2560 scope.go:117] "RemoveContainer" containerID="75ba25abfbdd5161953d0c444c3943fe6752c31a9c6dafef98cfdfde46d56a22"
Sep 17 14:14:16.488575 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:16.488561    2560 scope.go:117] "RemoveContainer" containerID="8a24814237a5257d743231cff70299f9ef4ec20ba2217e301a9d9bd80b62c834"
Sep 17 14:14:16.497360 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:16.497345    2560 scope.go:117] "RemoveContainer" containerID="75ba25abfbdd5161953d0c444c3943fe6752c31a9c6dafef98cfdfde46d56a22"
Sep 17 14:14:16.497705 ip-10-0-129-158 kubenswrapper[2560]: E0917 14:14:16.497685    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75ba25abfbdd5161953d0c444c3943fe6752c31a9c6dafef98cfdfde46d56a22\": container with ID starting with 75ba25abfbdd5161953d0c444c3943fe6752c31a9c6dafef98cfdfde46d56a22 not found: ID does not exist" containerID="75ba25abfbdd5161953d0c444c3943fe6752c31a9c6dafef98cfdfde46d56a22"
Sep 17 14:14:16.497780 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:16.497711    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75ba25abfbdd5161953d0c444c3943fe6752c31a9c6dafef98cfdfde46d56a22"} err="failed to get container status \"75ba25abfbdd5161953d0c444c3943fe6752c31a9c6dafef98cfdfde46d56a22\": rpc error: code = NotFound desc = could not find container \"75ba25abfbdd5161953d0c444c3943fe6752c31a9c6dafef98cfdfde46d56a22\": container with ID starting with 75ba25abfbdd5161953d0c444c3943fe6752c31a9c6dafef98cfdfde46d56a22 not found: ID does not exist"
Sep 17 14:14:16.497780 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:16.497726    2560 scope.go:117] "RemoveContainer" containerID="8a24814237a5257d743231cff70299f9ef4ec20ba2217e301a9d9bd80b62c834"
Sep 17 14:14:16.497995 ip-10-0-129-158 kubenswrapper[2560]: E0917 14:14:16.497982    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a24814237a5257d743231cff70299f9ef4ec20ba2217e301a9d9bd80b62c834\": container with ID starting with 8a24814237a5257d743231cff70299f9ef4ec20ba2217e301a9d9bd80b62c834 not found: ID does not exist" containerID="8a24814237a5257d743231cff70299f9ef4ec20ba2217e301a9d9bd80b62c834"
Sep 17 14:14:16.498040 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:16.497998    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a24814237a5257d743231cff70299f9ef4ec20ba2217e301a9d9bd80b62c834"} err="failed to get container status \"8a24814237a5257d743231cff70299f9ef4ec20ba2217e301a9d9bd80b62c834\": rpc error: code = NotFound desc = could not find container \"8a24814237a5257d743231cff70299f9ef4ec20ba2217e301a9d9bd80b62c834\": container with ID starting with 8a24814237a5257d743231cff70299f9ef4ec20ba2217e301a9d9bd80b62c834 not found: ID does not exist"
Sep 17 14:14:16.499688 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:16.499667    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94"]
Sep 17 14:14:16.503570 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:16.503552    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-9af6d23e-openshift-default-5dfb969684-zkt94"]
Sep 17 14:14:17.569870 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:17.569841    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59b5cd5f-71d1-44a3-acb1-a693b652c4c5" path="/var/lib/kubelet/pods/59b5cd5f-71d1-44a3-acb1-a693b652c4c5/volumes"
Sep 17 14:14:20.187105 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:20.187068    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:20.311283 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:20.311246    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:20.436414 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:20.436381    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:20.538293 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:20.538217    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:20.930910 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:20.930880    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:22.012894 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:22.012836    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-0458f0/autores-model-0458f0-kserve-6cd946794c-5p2zp" podUID="cfc3d9a0-e8cf-44d2-882b-3dc079266498" containerName="main" containerID="cri-o://9ad75b0ce23e7581dab9bed8731b23001cd35dfa86c62f29587bfc56dbb4772f" gracePeriod=45
Sep 17 14:14:22.267165 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:22.267103    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-0458f0/autores-model-0458f0-kserve-6cd946794c-5p2zp"
Sep 17 14:14:22.356646 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:22.356615    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfc3d9a0-e8cf-44d2-882b-3dc079266498-tls-certs\") pod \"cfc3d9a0-e8cf-44d2-882b-3dc079266498\" (UID: \"cfc3d9a0-e8cf-44d2-882b-3dc079266498\") "
Sep 17 14:14:22.356816 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:22.356656    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfc3d9a0-e8cf-44d2-882b-3dc079266498-home\") pod \"cfc3d9a0-e8cf-44d2-882b-3dc079266498\" (UID: \"cfc3d9a0-e8cf-44d2-882b-3dc079266498\") "
Sep 17 14:14:22.356816 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:22.356787    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfc3d9a0-e8cf-44d2-882b-3dc079266498-dshm\") pod \"cfc3d9a0-e8cf-44d2-882b-3dc079266498\" (UID: \"cfc3d9a0-e8cf-44d2-882b-3dc079266498\") "
Sep 17 14:14:22.356939 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:22.356897    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gjwpv\" (UniqueName: \"kubernetes.io/projected/cfc3d9a0-e8cf-44d2-882b-3dc079266498-kube-api-access-gjwpv\") pod \"cfc3d9a0-e8cf-44d2-882b-3dc079266498\" (UID: \"cfc3d9a0-e8cf-44d2-882b-3dc079266498\") "
Sep 17 14:14:22.356939 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:22.356913    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfc3d9a0-e8cf-44d2-882b-3dc079266498-home" (OuterVolumeSpecName: "home") pod "cfc3d9a0-e8cf-44d2-882b-3dc079266498" (UID: "cfc3d9a0-e8cf-44d2-882b-3dc079266498"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:14:22.356939 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:22.356933    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfc3d9a0-e8cf-44d2-882b-3dc079266498-model-cache\") pod \"cfc3d9a0-e8cf-44d2-882b-3dc079266498\" (UID: \"cfc3d9a0-e8cf-44d2-882b-3dc079266498\") "
Sep 17 14:14:22.357107 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:22.356964    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfc3d9a0-e8cf-44d2-882b-3dc079266498-tmp-dir\") pod \"cfc3d9a0-e8cf-44d2-882b-3dc079266498\" (UID: \"cfc3d9a0-e8cf-44d2-882b-3dc079266498\") "
Sep 17 14:14:22.357250 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:22.357217    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfc3d9a0-e8cf-44d2-882b-3dc079266498-model-cache" (OuterVolumeSpecName: "model-cache") pod "cfc3d9a0-e8cf-44d2-882b-3dc079266498" (UID: "cfc3d9a0-e8cf-44d2-882b-3dc079266498"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:14:22.357335 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:22.357256    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfc3d9a0-e8cf-44d2-882b-3dc079266498-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cfc3d9a0-e8cf-44d2-882b-3dc079266498" (UID: "cfc3d9a0-e8cf-44d2-882b-3dc079266498"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:14:22.357335 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:22.357310    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfc3d9a0-e8cf-44d2-882b-3dc079266498-model-cache\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:14:22.357335 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:22.357327    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfc3d9a0-e8cf-44d2-882b-3dc079266498-home\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:14:22.358852 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:22.358823    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfc3d9a0-e8cf-44d2-882b-3dc079266498-dshm" (OuterVolumeSpecName: "dshm") pod "cfc3d9a0-e8cf-44d2-882b-3dc079266498" (UID: "cfc3d9a0-e8cf-44d2-882b-3dc079266498"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:14:22.358973 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:22.358905    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfc3d9a0-e8cf-44d2-882b-3dc079266498-kube-api-access-gjwpv" (OuterVolumeSpecName: "kube-api-access-gjwpv") pod "cfc3d9a0-e8cf-44d2-882b-3dc079266498" (UID: "cfc3d9a0-e8cf-44d2-882b-3dc079266498"). InnerVolumeSpecName "kube-api-access-gjwpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:14:22.358973 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:22.358911    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfc3d9a0-e8cf-44d2-882b-3dc079266498-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cfc3d9a0-e8cf-44d2-882b-3dc079266498" (UID: "cfc3d9a0-e8cf-44d2-882b-3dc079266498"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:14:22.457906 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:22.457880    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfc3d9a0-e8cf-44d2-882b-3dc079266498-dshm\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:14:22.457906 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:22.457902    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjwpv\" (UniqueName: \"kubernetes.io/projected/cfc3d9a0-e8cf-44d2-882b-3dc079266498-kube-api-access-gjwpv\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:14:22.458067 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:22.457912    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfc3d9a0-e8cf-44d2-882b-3dc079266498-tmp-dir\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:14:22.458067 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:22.457921    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfc3d9a0-e8cf-44d2-882b-3dc079266498-tls-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:14:22.516892 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:22.516862    2560 generic.go:356] "Generic (PLEG): container finished" podID="cfc3d9a0-e8cf-44d2-882b-3dc079266498" containerID="9ad75b0ce23e7581dab9bed8731b23001cd35dfa86c62f29587bfc56dbb4772f" exitCode=0
Sep 17 14:14:22.517024 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:22.516906    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-0458f0/autores-model-0458f0-kserve-6cd946794c-5p2zp"
Sep 17 14:14:22.517024 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:22.516957    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-0458f0/autores-model-0458f0-kserve-6cd946794c-5p2zp" event={"ID":"cfc3d9a0-e8cf-44d2-882b-3dc079266498","Type":"ContainerDied","Data":"9ad75b0ce23e7581dab9bed8731b23001cd35dfa86c62f29587bfc56dbb4772f"}
Sep 17 14:14:22.517024 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:22.517002    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-0458f0/autores-model-0458f0-kserve-6cd946794c-5p2zp" event={"ID":"cfc3d9a0-e8cf-44d2-882b-3dc079266498","Type":"ContainerDied","Data":"c7acef45cc413cb23140305c81fbd5ba23e603147b365a3bc7f9ed51f0323090"}
Sep 17 14:14:22.517222 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:22.517026    2560 scope.go:117] "RemoveContainer" containerID="9ad75b0ce23e7581dab9bed8731b23001cd35dfa86c62f29587bfc56dbb4772f"
Sep 17 14:14:22.527104 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:22.527089    2560 scope.go:117] "RemoveContainer" containerID="9ad75b0ce23e7581dab9bed8731b23001cd35dfa86c62f29587bfc56dbb4772f"
Sep 17 14:14:22.527370 ip-10-0-129-158 kubenswrapper[2560]: E0917 14:14:22.527352    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ad75b0ce23e7581dab9bed8731b23001cd35dfa86c62f29587bfc56dbb4772f\": container with ID starting with 9ad75b0ce23e7581dab9bed8731b23001cd35dfa86c62f29587bfc56dbb4772f not found: ID does not exist" containerID="9ad75b0ce23e7581dab9bed8731b23001cd35dfa86c62f29587bfc56dbb4772f"
Sep 17 14:14:22.527411 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:22.527377    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ad75b0ce23e7581dab9bed8731b23001cd35dfa86c62f29587bfc56dbb4772f"} err="failed to get container status \"9ad75b0ce23e7581dab9bed8731b23001cd35dfa86c62f29587bfc56dbb4772f\": rpc error: code = NotFound desc = could not find container \"9ad75b0ce23e7581dab9bed8731b23001cd35dfa86c62f29587bfc56dbb4772f\": container with ID starting with 9ad75b0ce23e7581dab9bed8731b23001cd35dfa86c62f29587bfc56dbb4772f not found: ID does not exist"
Sep 17 14:14:22.538987 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:22.538958    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-0458f0/autores-model-0458f0-kserve-6cd946794c-5p2zp"]
Sep 17 14:14:22.549021 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:22.548988    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-0458f0/autores-model-0458f0-kserve-6cd946794c-5p2zp"]
Sep 17 14:14:23.569576 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:23.569547    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfc3d9a0-e8cf-44d2-882b-3dc079266498" path="/var/lib/kubelet/pods/cfc3d9a0-e8cf-44d2-882b-3dc079266498/volumes"
Sep 17 14:14:35.407893 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:35.407860    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:44.724440 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:44.724408    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-d9b17d/premium-w0-d9b17d-kserve-54647796d7-4r49f"]
Sep 17 14:14:44.724869 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:44.724854    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59b5cd5f-71d1-44a3-acb1-a693b652c4c5" containerName="istio-proxy"
Sep 17 14:14:44.724914 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:44.724871    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="59b5cd5f-71d1-44a3-acb1-a693b652c4c5" containerName="istio-proxy"
Sep 17 14:14:44.724914 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:44.724884    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db3fa3f8-47d9-44c5-ad4d-c26e218b08e7" containerName="revoke-keys"
Sep 17 14:14:44.724914 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:44.724890    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="db3fa3f8-47d9-44c5-ad4d-c26e218b08e7" containerName="revoke-keys"
Sep 17 14:14:44.724914 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:44.724909    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7e2e06f-693e-49f5-85ae-1f6768663911" containerName="payload-processing"
Sep 17 14:14:44.724914 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:44.724915    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7e2e06f-693e-49f5-85ae-1f6768663911" containerName="payload-processing"
Sep 17 14:14:44.725074 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:44.724921    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfc3d9a0-e8cf-44d2-882b-3dc079266498" containerName="main"
Sep 17 14:14:44.725074 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:44.724926    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfc3d9a0-e8cf-44d2-882b-3dc079266498" containerName="main"
Sep 17 14:14:44.725074 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:44.724991    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="59b5cd5f-71d1-44a3-acb1-a693b652c4c5" containerName="istio-proxy"
Sep 17 14:14:44.725074 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:44.724997    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="db3fa3f8-47d9-44c5-ad4d-c26e218b08e7" containerName="revoke-keys"
Sep 17 14:14:44.725074 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:44.725005    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="59b5cd5f-71d1-44a3-acb1-a693b652c4c5" containerName="istio-proxy"
Sep 17 14:14:44.725074 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:44.725014    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfc3d9a0-e8cf-44d2-882b-3dc079266498" containerName="main"
Sep 17 14:14:44.725074 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:44.725021    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7e2e06f-693e-49f5-85ae-1f6768663911" containerName="payload-processing"
Sep 17 14:14:44.729387 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:44.729369    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d9b17d/premium-w0-d9b17d-kserve-54647796d7-4r49f"
Sep 17 14:14:44.732858 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:44.732834    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-d9b17d\"/\"kube-root-ca.crt\""
Sep 17 14:14:44.733010 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:44.732846    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d9b17d\"/\"default-dockercfg-xhl2t\""
Sep 17 14:14:44.733225 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:44.732848    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-d9b17d\"/\"openshift-service-ca.crt\""
Sep 17 14:14:44.733373 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:44.732852    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d9b17d\"/\"premium-w0-d9b17d-kserve-self-signed-certs\""
Sep 17 14:14:44.737118 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:44.737094    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d9b17d/premium-w0-d9b17d-kserve-54647796d7-4r49f"]
Sep 17 14:14:44.752195 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:44.752161    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:44.846527 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:44.846496    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2809a4f4-2371-4513-8924-461ba4b6e5cf-tmp-dir\") pod \"premium-w0-d9b17d-kserve-54647796d7-4r49f\" (UID: \"2809a4f4-2371-4513-8924-461ba4b6e5cf\") " pod="e2e-models-e2e-worker-w0-d9b17d/premium-w0-d9b17d-kserve-54647796d7-4r49f"
Sep 17 14:14:44.846648 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:44.846564    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58qzg\" (UniqueName: \"kubernetes.io/projected/2809a4f4-2371-4513-8924-461ba4b6e5cf-kube-api-access-58qzg\") pod \"premium-w0-d9b17d-kserve-54647796d7-4r49f\" (UID: \"2809a4f4-2371-4513-8924-461ba4b6e5cf\") " pod="e2e-models-e2e-worker-w0-d9b17d/premium-w0-d9b17d-kserve-54647796d7-4r49f"
Sep 17 14:14:44.846648 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:44.846614    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2809a4f4-2371-4513-8924-461ba4b6e5cf-home\") pod \"premium-w0-d9b17d-kserve-54647796d7-4r49f\" (UID: \"2809a4f4-2371-4513-8924-461ba4b6e5cf\") " pod="e2e-models-e2e-worker-w0-d9b17d/premium-w0-d9b17d-kserve-54647796d7-4r49f"
Sep 17 14:14:44.846770 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:44.846682    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2809a4f4-2371-4513-8924-461ba4b6e5cf-model-cache\") pod \"premium-w0-d9b17d-kserve-54647796d7-4r49f\" (UID: \"2809a4f4-2371-4513-8924-461ba4b6e5cf\") " pod="e2e-models-e2e-worker-w0-d9b17d/premium-w0-d9b17d-kserve-54647796d7-4r49f"
Sep 17 14:14:44.846770 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:44.846727    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2809a4f4-2371-4513-8924-461ba4b6e5cf-tls-certs\") pod \"premium-w0-d9b17d-kserve-54647796d7-4r49f\" (UID: \"2809a4f4-2371-4513-8924-461ba4b6e5cf\") " pod="e2e-models-e2e-worker-w0-d9b17d/premium-w0-d9b17d-kserve-54647796d7-4r49f"
Sep 17 14:14:44.846770 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:44.846753    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2809a4f4-2371-4513-8924-461ba4b6e5cf-dshm\") pod \"premium-w0-d9b17d-kserve-54647796d7-4r49f\" (UID: \"2809a4f4-2371-4513-8924-461ba4b6e5cf\") " pod="e2e-models-e2e-worker-w0-d9b17d/premium-w0-d9b17d-kserve-54647796d7-4r49f"
Sep 17 14:14:44.948066 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:44.948033    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2809a4f4-2371-4513-8924-461ba4b6e5cf-tmp-dir\") pod \"premium-w0-d9b17d-kserve-54647796d7-4r49f\" (UID: \"2809a4f4-2371-4513-8924-461ba4b6e5cf\") " pod="e2e-models-e2e-worker-w0-d9b17d/premium-w0-d9b17d-kserve-54647796d7-4r49f"
Sep 17 14:14:44.948246 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:44.948090    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58qzg\" (UniqueName: \"kubernetes.io/projected/2809a4f4-2371-4513-8924-461ba4b6e5cf-kube-api-access-58qzg\") pod \"premium-w0-d9b17d-kserve-54647796d7-4r49f\" (UID: \"2809a4f4-2371-4513-8924-461ba4b6e5cf\") " pod="e2e-models-e2e-worker-w0-d9b17d/premium-w0-d9b17d-kserve-54647796d7-4r49f"
Sep 17 14:14:44.948246 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:44.948127    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2809a4f4-2371-4513-8924-461ba4b6e5cf-home\") pod \"premium-w0-d9b17d-kserve-54647796d7-4r49f\" (UID: \"2809a4f4-2371-4513-8924-461ba4b6e5cf\") " pod="e2e-models-e2e-worker-w0-d9b17d/premium-w0-d9b17d-kserve-54647796d7-4r49f"
Sep 17 14:14:44.948246 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:44.948162    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2809a4f4-2371-4513-8924-461ba4b6e5cf-model-cache\") pod \"premium-w0-d9b17d-kserve-54647796d7-4r49f\" (UID: \"2809a4f4-2371-4513-8924-461ba4b6e5cf\") " pod="e2e-models-e2e-worker-w0-d9b17d/premium-w0-d9b17d-kserve-54647796d7-4r49f"
Sep 17 14:14:44.948246 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:44.948194    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2809a4f4-2371-4513-8924-461ba4b6e5cf-tls-certs\") pod \"premium-w0-d9b17d-kserve-54647796d7-4r49f\" (UID: \"2809a4f4-2371-4513-8924-461ba4b6e5cf\") " pod="e2e-models-e2e-worker-w0-d9b17d/premium-w0-d9b17d-kserve-54647796d7-4r49f"
Sep 17 14:14:44.948246 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:44.948220    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2809a4f4-2371-4513-8924-461ba4b6e5cf-dshm\") pod \"premium-w0-d9b17d-kserve-54647796d7-4r49f\" (UID: \"2809a4f4-2371-4513-8924-461ba4b6e5cf\") " pod="e2e-models-e2e-worker-w0-d9b17d/premium-w0-d9b17d-kserve-54647796d7-4r49f"
Sep 17 14:14:44.948560 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:44.948535    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2809a4f4-2371-4513-8924-461ba4b6e5cf-tmp-dir\") pod \"premium-w0-d9b17d-kserve-54647796d7-4r49f\" (UID: \"2809a4f4-2371-4513-8924-461ba4b6e5cf\") " pod="e2e-models-e2e-worker-w0-d9b17d/premium-w0-d9b17d-kserve-54647796d7-4r49f"
Sep 17 14:14:44.948622 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:44.948598    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2809a4f4-2371-4513-8924-461ba4b6e5cf-model-cache\") pod \"premium-w0-d9b17d-kserve-54647796d7-4r49f\" (UID: \"2809a4f4-2371-4513-8924-461ba4b6e5cf\") " pod="e2e-models-e2e-worker-w0-d9b17d/premium-w0-d9b17d-kserve-54647796d7-4r49f"
Sep 17 14:14:44.948669 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:44.948646    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2809a4f4-2371-4513-8924-461ba4b6e5cf-home\") pod \"premium-w0-d9b17d-kserve-54647796d7-4r49f\" (UID: \"2809a4f4-2371-4513-8924-461ba4b6e5cf\") " pod="e2e-models-e2e-worker-w0-d9b17d/premium-w0-d9b17d-kserve-54647796d7-4r49f"
Sep 17 14:14:44.950441 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:44.950419    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2809a4f4-2371-4513-8924-461ba4b6e5cf-dshm\") pod \"premium-w0-d9b17d-kserve-54647796d7-4r49f\" (UID: \"2809a4f4-2371-4513-8924-461ba4b6e5cf\") " pod="e2e-models-e2e-worker-w0-d9b17d/premium-w0-d9b17d-kserve-54647796d7-4r49f"
Sep 17 14:14:44.950634 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:44.950616    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2809a4f4-2371-4513-8924-461ba4b6e5cf-tls-certs\") pod \"premium-w0-d9b17d-kserve-54647796d7-4r49f\" (UID: \"2809a4f4-2371-4513-8924-461ba4b6e5cf\") " pod="e2e-models-e2e-worker-w0-d9b17d/premium-w0-d9b17d-kserve-54647796d7-4r49f"
Sep 17 14:14:44.955824 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:44.955800    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58qzg\" (UniqueName: \"kubernetes.io/projected/2809a4f4-2371-4513-8924-461ba4b6e5cf-kube-api-access-58qzg\") pod \"premium-w0-d9b17d-kserve-54647796d7-4r49f\" (UID: \"2809a4f4-2371-4513-8924-461ba4b6e5cf\") " pod="e2e-models-e2e-worker-w0-d9b17d/premium-w0-d9b17d-kserve-54647796d7-4r49f"
Sep 17 14:14:45.021829 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:45.021763    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:45.039649 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:45.039624    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d9b17d/premium-w0-d9b17d-kserve-54647796d7-4r49f"
Sep 17 14:14:45.302091 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:45.302057    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:45.336549 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:45.336264    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d9b17d/premium-w0-d9b17d-kserve-54647796d7-4r49f"]
Sep 17 14:14:45.650118 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:45.650038    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d9b17d/premium-w0-d9b17d-kserve-54647796d7-4r49f" event={"ID":"2809a4f4-2371-4513-8924-461ba4b6e5cf","Type":"ContainerStarted","Data":"85438e8fed849f758191ac4225a745f60a7f9c5cc1c0124f14ab505afa347ab1"}
Sep 17 14:14:45.650118 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:45.650074    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d9b17d/premium-w0-d9b17d-kserve-54647796d7-4r49f" event={"ID":"2809a4f4-2371-4513-8924-461ba4b6e5cf","Type":"ContainerStarted","Data":"01ae4de72d17a88d455caebad13c9d7133c415f623a26431b07c27a7f933a5a1"}
Sep 17 14:14:45.671340 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:45.671293    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-d9b17d/premium-w0-d9b17d-kserve-54647796d7-4r49f" podStartSLOduration=1.671279642 podStartE2EDuration="1.671279642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:14:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:14:45.668658306 +0000 UTC m=+1154.610801953" watchObservedRunningTime="2026-09-17 14:14:45.671279642 +0000 UTC m=+1154.613423322"
Sep 17 14:14:45.811516 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:45.811480    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:46.165108 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:46.165078    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s"]
Sep 17 14:14:46.165605 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:46.165590    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59b5cd5f-71d1-44a3-acb1-a693b652c4c5" containerName="istio-proxy"
Sep 17 14:14:46.165648 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:46.165607    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="59b5cd5f-71d1-44a3-acb1-a693b652c4c5" containerName="istio-proxy"
Sep 17 14:14:46.168850 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:46.168833    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s"
Sep 17 14:14:46.171823 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:46.171799    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-767170-openshift-default-dockercfg-8kdmc\""
Sep 17 14:14:46.188705 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:46.188675    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:46.190131 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:46.190112    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s"]
Sep 17 14:14:46.260369 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:46.260343    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ef3eba7-9ba6-46e2-aa43-cfa423635b70-istiod-ca-cert\") pod \"e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s\" (UID: \"7ef3eba7-9ba6-46e2-aa43-cfa423635b70\") " pod="openshift-ingress/e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s"
Sep 17 14:14:46.260558 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:46.260377    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2cs6\" (UniqueName: \"kubernetes.io/projected/7ef3eba7-9ba6-46e2-aa43-cfa423635b70-kube-api-access-w2cs6\") pod \"e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s\" (UID: \"7ef3eba7-9ba6-46e2-aa43-cfa423635b70\") " pod="openshift-ingress/e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s"
Sep 17 14:14:46.260558 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:46.260408    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ef3eba7-9ba6-46e2-aa43-cfa423635b70-workload-certs\") pod \"e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s\" (UID: \"7ef3eba7-9ba6-46e2-aa43-cfa423635b70\") " pod="openshift-ingress/e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s"
Sep 17 14:14:46.260558 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:46.260436    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ef3eba7-9ba6-46e2-aa43-cfa423635b70-istio-data\") pod \"e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s\" (UID: \"7ef3eba7-9ba6-46e2-aa43-cfa423635b70\") " pod="openshift-ingress/e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s"
Sep 17 14:14:46.260558 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:46.260501    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ef3eba7-9ba6-46e2-aa43-cfa423635b70-credential-socket\") pod \"e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s\" (UID: \"7ef3eba7-9ba6-46e2-aa43-cfa423635b70\") " pod="openshift-ingress/e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s"
Sep 17 14:14:46.260558 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:46.260539    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ef3eba7-9ba6-46e2-aa43-cfa423635b70-workload-socket\") pod \"e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s\" (UID: \"7ef3eba7-9ba6-46e2-aa43-cfa423635b70\") " pod="openshift-ingress/e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s"
Sep 17 14:14:46.260737 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:46.260583    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ef3eba7-9ba6-46e2-aa43-cfa423635b70-istio-token\") pod \"e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s\" (UID: \"7ef3eba7-9ba6-46e2-aa43-cfa423635b70\") " pod="openshift-ingress/e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s"
Sep 17 14:14:46.260737 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:46.260609    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ef3eba7-9ba6-46e2-aa43-cfa423635b70-istio-envoy\") pod \"e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s\" (UID: \"7ef3eba7-9ba6-46e2-aa43-cfa423635b70\") " pod="openshift-ingress/e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s"
Sep 17 14:14:46.260737 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:46.260626    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ef3eba7-9ba6-46e2-aa43-cfa423635b70-istio-podinfo\") pod \"e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s\" (UID: \"7ef3eba7-9ba6-46e2-aa43-cfa423635b70\") " pod="openshift-ingress/e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s"
Sep 17 14:14:46.336568 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:46.336526    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:46.361884 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:46.361849    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ef3eba7-9ba6-46e2-aa43-cfa423635b70-istiod-ca-cert\") pod \"e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s\" (UID: \"7ef3eba7-9ba6-46e2-aa43-cfa423635b70\") " pod="openshift-ingress/e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s"
Sep 17 14:14:46.362036 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:46.361906    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2cs6\" (UniqueName: \"kubernetes.io/projected/7ef3eba7-9ba6-46e2-aa43-cfa423635b70-kube-api-access-w2cs6\") pod \"e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s\" (UID: \"7ef3eba7-9ba6-46e2-aa43-cfa423635b70\") " pod="openshift-ingress/e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s"
Sep 17 14:14:46.362036 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:46.361954    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ef3eba7-9ba6-46e2-aa43-cfa423635b70-workload-certs\") pod \"e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s\" (UID: \"7ef3eba7-9ba6-46e2-aa43-cfa423635b70\") " pod="openshift-ingress/e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s"
Sep 17 14:14:46.362036 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:46.361988    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ef3eba7-9ba6-46e2-aa43-cfa423635b70-istio-data\") pod \"e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s\" (UID: \"7ef3eba7-9ba6-46e2-aa43-cfa423635b70\") " pod="openshift-ingress/e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s"
Sep 17 14:14:46.362221 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:46.362033    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ef3eba7-9ba6-46e2-aa43-cfa423635b70-credential-socket\") pod \"e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s\" (UID: \"7ef3eba7-9ba6-46e2-aa43-cfa423635b70\") " pod="openshift-ingress/e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s"
Sep 17 14:14:46.362221 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:46.362095    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ef3eba7-9ba6-46e2-aa43-cfa423635b70-workload-socket\") pod \"e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s\" (UID: \"7ef3eba7-9ba6-46e2-aa43-cfa423635b70\") " pod="openshift-ingress/e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s"
Sep 17 14:14:46.362221 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:46.362138    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ef3eba7-9ba6-46e2-aa43-cfa423635b70-istio-token\") pod \"e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s\" (UID: \"7ef3eba7-9ba6-46e2-aa43-cfa423635b70\") " pod="openshift-ingress/e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s"
Sep 17 14:14:46.362221 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:46.362177    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ef3eba7-9ba6-46e2-aa43-cfa423635b70-istio-envoy\") pod \"e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s\" (UID: \"7ef3eba7-9ba6-46e2-aa43-cfa423635b70\") " pod="openshift-ingress/e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s"
Sep 17 14:14:46.362221 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:46.362203    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ef3eba7-9ba6-46e2-aa43-cfa423635b70-istio-podinfo\") pod \"e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s\" (UID: \"7ef3eba7-9ba6-46e2-aa43-cfa423635b70\") " pod="openshift-ingress/e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s"
Sep 17 14:14:46.362700 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:46.362672    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ef3eba7-9ba6-46e2-aa43-cfa423635b70-istiod-ca-cert\") pod \"e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s\" (UID: \"7ef3eba7-9ba6-46e2-aa43-cfa423635b70\") " pod="openshift-ingress/e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s"
Sep 17 14:14:46.362931 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:46.362791    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ef3eba7-9ba6-46e2-aa43-cfa423635b70-workload-certs\") pod \"e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s\" (UID: \"7ef3eba7-9ba6-46e2-aa43-cfa423635b70\") " pod="openshift-ingress/e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s"
Sep 17 14:14:46.363013 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:46.362987    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ef3eba7-9ba6-46e2-aa43-cfa423635b70-istio-data\") pod \"e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s\" (UID: \"7ef3eba7-9ba6-46e2-aa43-cfa423635b70\") " pod="openshift-ingress/e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s"
Sep 17 14:14:46.363293 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:46.363245    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ef3eba7-9ba6-46e2-aa43-cfa423635b70-workload-socket\") pod \"e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s\" (UID: \"7ef3eba7-9ba6-46e2-aa43-cfa423635b70\") " pod="openshift-ingress/e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s"
Sep 17 14:14:46.363493 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:46.363324    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ef3eba7-9ba6-46e2-aa43-cfa423635b70-credential-socket\") pod \"e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s\" (UID: \"7ef3eba7-9ba6-46e2-aa43-cfa423635b70\") " pod="openshift-ingress/e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s"
Sep 17 14:14:46.365926 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:46.365898    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ef3eba7-9ba6-46e2-aa43-cfa423635b70-istio-envoy\") pod \"e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s\" (UID: \"7ef3eba7-9ba6-46e2-aa43-cfa423635b70\") " pod="openshift-ingress/e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s"
Sep 17 14:14:46.366012 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:46.365977    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ef3eba7-9ba6-46e2-aa43-cfa423635b70-istio-podinfo\") pod \"e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s\" (UID: \"7ef3eba7-9ba6-46e2-aa43-cfa423635b70\") " pod="openshift-ingress/e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s"
Sep 17 14:14:46.370892 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:46.370871    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ef3eba7-9ba6-46e2-aa43-cfa423635b70-istio-token\") pod \"e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s\" (UID: \"7ef3eba7-9ba6-46e2-aa43-cfa423635b70\") " pod="openshift-ingress/e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s"
Sep 17 14:14:46.371978 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:46.371960    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2cs6\" (UniqueName: \"kubernetes.io/projected/7ef3eba7-9ba6-46e2-aa43-cfa423635b70-kube-api-access-w2cs6\") pod \"e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s\" (UID: \"7ef3eba7-9ba6-46e2-aa43-cfa423635b70\") " pod="openshift-ingress/e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s"
Sep 17 14:14:46.479824 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:46.479745    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s"
Sep 17 14:14:46.757227 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:46.757198    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s"]
Sep 17 14:14:46.758606 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:14:46.758581    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ef3eba7_9ba6_46e2_aa43_cfa423635b70.slice/crio-62587722967d3ef3e6ef2f4de09742a859bcff5531dcfcee25bcf88793a02a60 WatchSource:0}: Error finding container 62587722967d3ef3e6ef2f4de09742a859bcff5531dcfcee25bcf88793a02a60: Status 404 returned error can't find the container with id 62587722967d3ef3e6ef2f4de09742a859bcff5531dcfcee25bcf88793a02a60
Sep 17 14:14:46.760777 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:46.760745    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:14:46.760861 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:46.760809    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:14:46.760861 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:46.760842    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:14:46.908154 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:46.908123    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:47.663163 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:47.663127    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s" event={"ID":"7ef3eba7-9ba6-46e2-aa43-cfa423635b70","Type":"ContainerStarted","Data":"c978ec5b559db39a4409a204f9cf6cf3e92649464791e8bcbc01c3575e11fc7f"}
Sep 17 14:14:47.663348 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:47.663179    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s" event={"ID":"7ef3eba7-9ba6-46e2-aa43-cfa423635b70","Type":"ContainerStarted","Data":"62587722967d3ef3e6ef2f4de09742a859bcff5531dcfcee25bcf88793a02a60"}
Sep 17 14:14:47.685059 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:47.685014    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s" podStartSLOduration=1.685000279 podStartE2EDuration="1.685000279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:14:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:14:47.683126667 +0000 UTC m=+1156.625270324" watchObservedRunningTime="2026-09-17 14:14:47.685000279 +0000 UTC m=+1156.627143924"
Sep 17 14:14:48.480536 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:48.480503    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s"
Sep 17 14:14:48.486268 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:48.486244    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s"
Sep 17 14:14:48.669556 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:48.669521    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s"
Sep 17 14:14:48.670537 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:48.670516    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s"
Sep 17 14:14:48.959208 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:48.959180    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:49.637734 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:49.637699    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:49.757080 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:49.757040    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:49.998804 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:49.998717    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:51.176165 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:51.176128    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:52.102924 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:52.102893    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:53.128435 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:53.128404    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:53.977340 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:53.977308    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:54.044998 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:54.044928    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:54.228429 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:54.228335    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:54.537504 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:54.537386    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:54.656664 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:54.656629    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:55.040436 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:55.040398    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d9b17d/premium-w0-d9b17d-kserve-54647796d7-4r49f"
Sep 17 14:14:55.040632 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:55.040513    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-d9b17d/premium-w0-d9b17d-kserve-54647796d7-4r49f"
Sep 17 14:14:55.052579 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:55.052558    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-d9b17d/premium-w0-d9b17d-kserve-54647796d7-4r49f"
Sep 17 14:14:55.726858 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:55.726828    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d9b17d/premium-w0-d9b17d-kserve-54647796d7-4r49f"
Sep 17 14:14:56.120627 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:56.120596    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:56.229200 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:56.229168    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:56.311825 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:14:56.311791    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:15:00.156221 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:00.155859    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d9b17d-29827575-44kg8"]
Sep 17 14:15:00.159755 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:00.159736    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-079b38-29827575-gp6bp"]
Sep 17 14:15:00.159880 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:00.159858    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d9b17d-29827575-44kg8"
Sep 17 14:15:00.162533 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:00.162505    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rp9ww\""
Sep 17 14:15:00.163242 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:00.163227    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29827575-cfsg7"]
Sep 17 14:15:00.163358 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:00.163342    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-079b38-29827575-gp6bp"
Sep 17 14:15:00.171521 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:00.169604    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-96cfc6-29827575-4t4k8"]
Sep 17 14:15:00.171521 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:00.171279    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29827575-cfsg7"
Sep 17 14:15:00.175636 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:00.175612    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d9b17d-29827575-44kg8"]
Sep 17 14:15:00.175781 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:00.175752    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-96cfc6-29827575-4t4k8"
Sep 17 14:15:00.179357 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:00.179334    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-96cfc6-29827575-4t4k8"]
Sep 17 14:15:00.182442 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:00.182409    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d9b17d-29827575-44kg8"
Sep 17 14:15:00.183658 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:00.183638    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-079b38-29827575-gp6bp"]
Sep 17 14:15:00.187853 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:00.187830    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29827575-cfsg7"]
Sep 17 14:15:00.192057 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:00.191853    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-079b38-29827575-gp6bp"
Sep 17 14:15:00.198295 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:00.198277    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29827575-cfsg7"
Sep 17 14:15:00.243673 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:00.243649    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-96cfc6-29827575-4t4k8"
Sep 17 14:15:00.486549 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:00.486523    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-079b38-29827575-gp6bp"]
Sep 17 14:15:00.487042 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:15:00.487017    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf79912d1_de41_45e4_89b6_0db0e97c50c9.slice/crio-73c4e5dd3983505ab25d55acb7f6f2e44792fd247f55d6ee9254550dbb543718 WatchSource:0}: Error finding container 73c4e5dd3983505ab25d55acb7f6f2e44792fd247f55d6ee9254550dbb543718: Status 404 returned error can't find the container with id 73c4e5dd3983505ab25d55acb7f6f2e44792fd247f55d6ee9254550dbb543718
Sep 17 14:15:00.487914 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:15:00.487824    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4341743e_5216_4b8d_9310_78acf4a761b4.slice/crio-86283396bfb17c7384e64c2b42c4b974f39b3fa4232692e385035c5120997e3d WatchSource:0}: Error finding container 86283396bfb17c7384e64c2b42c4b974f39b3fa4232692e385035c5120997e3d: Status 404 returned error can't find the container with id 86283396bfb17c7384e64c2b42c4b974f39b3fa4232692e385035c5120997e3d
Sep 17 14:15:00.489907 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:00.489884    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d9b17d-29827575-44kg8"]
Sep 17 14:15:00.511323 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:15:00.511296    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3792946f_bb55_422c_9182_9be8a911a6cd.slice/crio-aed2287c6ae1709ee3d9e05259d58637f49048c3953084c8d9c2bb4b87329e89 WatchSource:0}: Error finding container aed2287c6ae1709ee3d9e05259d58637f49048c3953084c8d9c2bb4b87329e89: Status 404 returned error can't find the container with id aed2287c6ae1709ee3d9e05259d58637f49048c3953084c8d9c2bb4b87329e89
Sep 17 14:15:00.511621 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:00.511601    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29827575-cfsg7"]
Sep 17 14:15:00.545705 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:00.545684    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-96cfc6-29827575-4t4k8"]
Sep 17 14:15:00.546865 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:15:00.546839    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8559995e_b946_4a42_b149_eda43a4c0e5a.slice/crio-df62eadec2190d106fcb9045eeeabd12b8cdd1237b3b511241b27524dc437e57 WatchSource:0}: Error finding container df62eadec2190d106fcb9045eeeabd12b8cdd1237b3b511241b27524dc437e57: Status 404 returned error can't find the container with id df62eadec2190d106fcb9045eeeabd12b8cdd1237b3b511241b27524dc437e57
Sep 17 14:15:00.747298 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:00.747238    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-079b38-29827575-gp6bp" event={"ID":"f79912d1-de41-45e4-89b6-0db0e97c50c9","Type":"ContainerStarted","Data":"73c4e5dd3983505ab25d55acb7f6f2e44792fd247f55d6ee9254550dbb543718"}
Sep 17 14:15:00.748527 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:00.748503    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-96cfc6-29827575-4t4k8" event={"ID":"8559995e-b946-4a42-b149-eda43a4c0e5a","Type":"ContainerStarted","Data":"df62eadec2190d106fcb9045eeeabd12b8cdd1237b3b511241b27524dc437e57"}
Sep 17 14:15:00.749602 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:00.749572    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d9b17d-29827575-44kg8" event={"ID":"4341743e-5216-4b8d-9310-78acf4a761b4","Type":"ContainerStarted","Data":"86283396bfb17c7384e64c2b42c4b974f39b3fa4232692e385035c5120997e3d"}
Sep 17 14:15:00.750697 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:00.750676    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29827575-cfsg7" event={"ID":"3792946f-bb55-422c-9182-9be8a911a6cd","Type":"ContainerStarted","Data":"aed2287c6ae1709ee3d9e05259d58637f49048c3953084c8d9c2bb4b87329e89"}
Sep 17 14:15:01.758611 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:01.758524    2560 generic.go:356] "Generic (PLEG): container finished" podID="f79912d1-de41-45e4-89b6-0db0e97c50c9" containerID="d1a9bba4bbd6ef0fdfa136e61e7cd56764edd5da30c7bc56359cec64d4fb1279" exitCode=0
Sep 17 14:15:01.759022 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:01.758612    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-079b38-29827575-gp6bp" event={"ID":"f79912d1-de41-45e4-89b6-0db0e97c50c9","Type":"ContainerDied","Data":"d1a9bba4bbd6ef0fdfa136e61e7cd56764edd5da30c7bc56359cec64d4fb1279"}
Sep 17 14:15:01.760309 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:01.760288    2560 generic.go:356] "Generic (PLEG): container finished" podID="8559995e-b946-4a42-b149-eda43a4c0e5a" containerID="3f4aab5fc7e782791aa422008c156fba1370a65755b90991183c40aea26a8738" exitCode=0
Sep 17 14:15:01.760409 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:01.760364    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-96cfc6-29827575-4t4k8" event={"ID":"8559995e-b946-4a42-b149-eda43a4c0e5a","Type":"ContainerDied","Data":"3f4aab5fc7e782791aa422008c156fba1370a65755b90991183c40aea26a8738"}
Sep 17 14:15:01.761815 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:01.761788    2560 generic.go:356] "Generic (PLEG): container finished" podID="4341743e-5216-4b8d-9310-78acf4a761b4" containerID="7c018fc05ddf363b4d5de88bba3bf168ba78ab9eb48226de81bdfb18c8cf996f" exitCode=0
Sep 17 14:15:01.761900 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:01.761814    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d9b17d-29827575-44kg8" event={"ID":"4341743e-5216-4b8d-9310-78acf4a761b4","Type":"ContainerDied","Data":"7c018fc05ddf363b4d5de88bba3bf168ba78ab9eb48226de81bdfb18c8cf996f"}
Sep 17 14:15:01.763356 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:01.763340    2560 generic.go:356] "Generic (PLEG): container finished" podID="3792946f-bb55-422c-9182-9be8a911a6cd" containerID="9d884f265fcd4a6eff6cd710c4fb8354862d18ea6684896c75909edce69db225" exitCode=0
Sep 17 14:15:01.763430 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:01.763381    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29827575-cfsg7" event={"ID":"3792946f-bb55-422c-9182-9be8a911a6cd","Type":"ContainerDied","Data":"9d884f265fcd4a6eff6cd710c4fb8354862d18ea6684896c75909edce69db225"}
Sep 17 14:15:02.916182 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:02.916160    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d9b17d-29827575-44kg8"
Sep 17 14:15:02.957819 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:02.957796    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29827575-cfsg7"
Sep 17 14:15:02.994366 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:02.994338    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-079b38-29827575-gp6bp"
Sep 17 14:15:02.997613 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:02.997595    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-96cfc6-29827575-4t4k8"
Sep 17 14:15:03.784170 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:03.784145    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d9b17d-29827575-44kg8"
Sep 17 14:15:03.784326 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:03.784144    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d9b17d-29827575-44kg8" event={"ID":"4341743e-5216-4b8d-9310-78acf4a761b4","Type":"ContainerDied","Data":"86283396bfb17c7384e64c2b42c4b974f39b3fa4232692e385035c5120997e3d"}
Sep 17 14:15:03.784326 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:03.784274    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86283396bfb17c7384e64c2b42c4b974f39b3fa4232692e385035c5120997e3d"
Sep 17 14:15:03.785521 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:03.785492    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29827575-cfsg7" event={"ID":"3792946f-bb55-422c-9182-9be8a911a6cd","Type":"ContainerDied","Data":"aed2287c6ae1709ee3d9e05259d58637f49048c3953084c8d9c2bb4b87329e89"}
Sep 17 14:15:03.785642 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:03.785547    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aed2287c6ae1709ee3d9e05259d58637f49048c3953084c8d9c2bb4b87329e89"
Sep 17 14:15:03.785642 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:03.785499    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29827575-cfsg7"
Sep 17 14:15:03.787071 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:03.787053    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-079b38-29827575-gp6bp"
Sep 17 14:15:03.787178 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:03.787077    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-079b38-29827575-gp6bp" event={"ID":"f79912d1-de41-45e4-89b6-0db0e97c50c9","Type":"ContainerDied","Data":"73c4e5dd3983505ab25d55acb7f6f2e44792fd247f55d6ee9254550dbb543718"}
Sep 17 14:15:03.787178 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:03.787116    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73c4e5dd3983505ab25d55acb7f6f2e44792fd247f55d6ee9254550dbb543718"
Sep 17 14:15:03.788531 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:03.788513    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-96cfc6-29827575-4t4k8" event={"ID":"8559995e-b946-4a42-b149-eda43a4c0e5a","Type":"ContainerDied","Data":"df62eadec2190d106fcb9045eeeabd12b8cdd1237b3b511241b27524dc437e57"}
Sep 17 14:15:03.788611 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:03.788545    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df62eadec2190d106fcb9045eeeabd12b8cdd1237b3b511241b27524dc437e57"
Sep 17 14:15:03.788611 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:03.788547    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-96cfc6-29827575-4t4k8"
Sep 17 14:15:12.191850 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:12.191754    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:15:15.879021 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:15.878987    2560 generic.go:356] "Generic (PLEG): container finished" podID="7ef3eba7-9ba6-46e2-aa43-cfa423635b70" containerID="c978ec5b559db39a4409a204f9cf6cf3e92649464791e8bcbc01c3575e11fc7f" exitCode=0
Sep 17 14:15:15.879433 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:15.879060    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s" event={"ID":"7ef3eba7-9ba6-46e2-aa43-cfa423635b70","Type":"ContainerDied","Data":"c978ec5b559db39a4409a204f9cf6cf3e92649464791e8bcbc01c3575e11fc7f"}
Sep 17 14:15:15.879433 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:15.879429    2560 scope.go:117] "RemoveContainer" containerID="c978ec5b559db39a4409a204f9cf6cf3e92649464791e8bcbc01c3575e11fc7f"
Sep 17 14:15:15.880552 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:15.880526    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:15:15.880632 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:15.880589    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:15:15.880632 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:15.880618    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:15:16.339661 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:16.339632    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:15:16.892139 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:16.892105    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s" event={"ID":"7ef3eba7-9ba6-46e2-aa43-cfa423635b70","Type":"ContainerStarted","Data":"2c3dd644b4cf8ff8db3452bb2f51d985342bb383464e0176d1c325c13f8ce683"}
Sep 17 14:15:16.941592 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:16.941558    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:15:17.157794 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:17.157682    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:15:17.480475 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:17.480356    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s"
Sep 17 14:15:17.481437 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:17.481410    2560 patch_prober.go:28] interesting pod/e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Sep 17 14:15:17.481604 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:17.481493    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s" podUID="7ef3eba7-9ba6-46e2-aa43-cfa423635b70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Sep 17 14:15:18.112357 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:18.112329    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:15:18.480910 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:18.480812    2560 patch_prober.go:28] interesting pod/e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Sep 17 14:15:18.481178 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:18.481149    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s" podUID="7ef3eba7-9ba6-46e2-aa43-cfa423635b70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Sep 17 14:15:18.670352 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:18.670313    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s"
Sep 17 14:15:19.106437 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:19.106404    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:15:19.481020 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:19.480923    2560 patch_prober.go:28] interesting pod/e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Sep 17 14:15:19.481020 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:19.480995    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s" podUID="7ef3eba7-9ba6-46e2-aa43-cfa423635b70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Sep 17 14:15:19.697621 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:19.697585    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:15:20.010490 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:20.010433    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:15:20.481244 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:20.481202    2560 patch_prober.go:28] interesting pod/e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Sep 17 14:15:20.481806 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:20.481773    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s" podUID="7ef3eba7-9ba6-46e2-aa43-cfa423635b70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Sep 17 14:15:20.799578 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:20.799530    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:15:20.899482 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:20.899432    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:15:20.999518 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:20.999483    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:15:21.099391 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:21.099307    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:15:21.480605 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:21.480513    2560 patch_prober.go:28] interesting pod/e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Sep 17 14:15:21.480605 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:21.480587    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s" podUID="7ef3eba7-9ba6-46e2-aa43-cfa423635b70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Sep 17 14:15:22.483844 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:22.483813    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s"
Sep 17 14:15:22.484764 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:22.484747    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s"
Sep 17 14:15:30.724551 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:30.724510    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:15:31.527948 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:31.527911    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:15:32.344701 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:32.344667    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:15:39.141153 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:39.141121    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:15:39.245518 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:39.245479    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:15:39.341002 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:39.340969    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:15:39.441610 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:39.441517    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:15:40.142991 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:40.142959    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:15:40.241581 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:40.241547    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:15:40.342531 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:40.342494    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:15:41.883132 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:41.883103    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:15:42.715279 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:42.715247    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-fe6ee5/test-model-fe6ee5-kserve-7774b7d84f-rn4bs"]
Sep 17 14:15:42.715944 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:42.715928    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4341743e-5216-4b8d-9310-78acf4a761b4" containerName="cleanup"
Sep 17 14:15:42.716007 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:42.715947    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4341743e-5216-4b8d-9310-78acf4a761b4" containerName="cleanup"
Sep 17 14:15:42.716007 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:42.715962    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3792946f-bb55-422c-9182-9be8a911a6cd" containerName="cleanup"
Sep 17 14:15:42.716007 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:42.715971    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3792946f-bb55-422c-9182-9be8a911a6cd" containerName="cleanup"
Sep 17 14:15:42.716007 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:42.715990    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8559995e-b946-4a42-b149-eda43a4c0e5a" containerName="cleanup"
Sep 17 14:15:42.716007 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:42.715999    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8559995e-b946-4a42-b149-eda43a4c0e5a" containerName="cleanup"
Sep 17 14:15:42.716173 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:42.716019    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f79912d1-de41-45e4-89b6-0db0e97c50c9" containerName="cleanup"
Sep 17 14:15:42.716173 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:42.716027    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f79912d1-de41-45e4-89b6-0db0e97c50c9" containerName="cleanup"
Sep 17 14:15:42.716173 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:42.716128    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4341743e-5216-4b8d-9310-78acf4a761b4" containerName="cleanup"
Sep 17 14:15:42.716173 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:42.716139    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f79912d1-de41-45e4-89b6-0db0e97c50c9" containerName="cleanup"
Sep 17 14:15:42.716173 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:42.716147    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="8559995e-b946-4a42-b149-eda43a4c0e5a" containerName="cleanup"
Sep 17 14:15:42.716173 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:42.716157    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3792946f-bb55-422c-9182-9be8a911a6cd" containerName="cleanup"
Sep 17 14:15:42.720085 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:42.720058    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fe6ee5/test-model-fe6ee5-kserve-7774b7d84f-rn4bs"
Sep 17 14:15:42.722640 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:42.722616    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcf756dc-ea28-4e48-b015-704955375715-home\") pod \"test-model-fe6ee5-kserve-7774b7d84f-rn4bs\" (UID: \"fcf756dc-ea28-4e48-b015-704955375715\") " pod="ai-tenant-e2e-inf-b-fe6ee5/test-model-fe6ee5-kserve-7774b7d84f-rn4bs"
Sep 17 14:15:42.722754 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:42.722655    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zzf5\" (UniqueName: \"kubernetes.io/projected/fcf756dc-ea28-4e48-b015-704955375715-kube-api-access-6zzf5\") pod \"test-model-fe6ee5-kserve-7774b7d84f-rn4bs\" (UID: \"fcf756dc-ea28-4e48-b015-704955375715\") " pod="ai-tenant-e2e-inf-b-fe6ee5/test-model-fe6ee5-kserve-7774b7d84f-rn4bs"
Sep 17 14:15:42.722754 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:42.722702    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcf756dc-ea28-4e48-b015-704955375715-tls-certs\") pod \"test-model-fe6ee5-kserve-7774b7d84f-rn4bs\" (UID: \"fcf756dc-ea28-4e48-b015-704955375715\") " pod="ai-tenant-e2e-inf-b-fe6ee5/test-model-fe6ee5-kserve-7774b7d84f-rn4bs"
Sep 17 14:15:42.722874 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:42.722788    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcf756dc-ea28-4e48-b015-704955375715-dshm\") pod \"test-model-fe6ee5-kserve-7774b7d84f-rn4bs\" (UID: \"fcf756dc-ea28-4e48-b015-704955375715\") " pod="ai-tenant-e2e-inf-b-fe6ee5/test-model-fe6ee5-kserve-7774b7d84f-rn4bs"
Sep 17 14:15:42.722874 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:42.722814    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcf756dc-ea28-4e48-b015-704955375715-model-cache\") pod \"test-model-fe6ee5-kserve-7774b7d84f-rn4bs\" (UID: \"fcf756dc-ea28-4e48-b015-704955375715\") " pod="ai-tenant-e2e-inf-b-fe6ee5/test-model-fe6ee5-kserve-7774b7d84f-rn4bs"
Sep 17 14:15:42.722874 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:42.722851    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcf756dc-ea28-4e48-b015-704955375715-tmp-dir\") pod \"test-model-fe6ee5-kserve-7774b7d84f-rn4bs\" (UID: \"fcf756dc-ea28-4e48-b015-704955375715\") " pod="ai-tenant-e2e-inf-b-fe6ee5/test-model-fe6ee5-kserve-7774b7d84f-rn4bs"
Sep 17 14:15:42.723044 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:42.723024    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-fe6ee5\"/\"kube-root-ca.crt\""
Sep 17 14:15:42.723094 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:42.723047    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-fe6ee5\"/\"openshift-service-ca.crt\""
Sep 17 14:15:42.723139 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:42.723024    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-fe6ee5\"/\"default-dockercfg-22n2z\""
Sep 17 14:15:42.723193 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:42.723168    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-fe6ee5\"/\"test-model-fe6ee5-kserve-self-signed-certs\""
Sep 17 14:15:42.728034 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:42.728010    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-fe6ee5/test-model-fe6ee5-kserve-7774b7d84f-rn4bs"]
Sep 17 14:15:42.743497 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:42.743450    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:15:42.824576 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:42.824537    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcf756dc-ea28-4e48-b015-704955375715-home\") pod \"test-model-fe6ee5-kserve-7774b7d84f-rn4bs\" (UID: \"fcf756dc-ea28-4e48-b015-704955375715\") " pod="ai-tenant-e2e-inf-b-fe6ee5/test-model-fe6ee5-kserve-7774b7d84f-rn4bs"
Sep 17 14:15:42.824576 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:42.824593    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zzf5\" (UniqueName: \"kubernetes.io/projected/fcf756dc-ea28-4e48-b015-704955375715-kube-api-access-6zzf5\") pod \"test-model-fe6ee5-kserve-7774b7d84f-rn4bs\" (UID: \"fcf756dc-ea28-4e48-b015-704955375715\") " pod="ai-tenant-e2e-inf-b-fe6ee5/test-model-fe6ee5-kserve-7774b7d84f-rn4bs"
Sep 17 14:15:42.824839 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:42.824643    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcf756dc-ea28-4e48-b015-704955375715-tls-certs\") pod \"test-model-fe6ee5-kserve-7774b7d84f-rn4bs\" (UID: \"fcf756dc-ea28-4e48-b015-704955375715\") " pod="ai-tenant-e2e-inf-b-fe6ee5/test-model-fe6ee5-kserve-7774b7d84f-rn4bs"
Sep 17 14:15:42.824839 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:42.824739    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcf756dc-ea28-4e48-b015-704955375715-dshm\") pod \"test-model-fe6ee5-kserve-7774b7d84f-rn4bs\" (UID: \"fcf756dc-ea28-4e48-b015-704955375715\") " pod="ai-tenant-e2e-inf-b-fe6ee5/test-model-fe6ee5-kserve-7774b7d84f-rn4bs"
Sep 17 14:15:42.824839 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:42.824763    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcf756dc-ea28-4e48-b015-704955375715-model-cache\") pod \"test-model-fe6ee5-kserve-7774b7d84f-rn4bs\" (UID: \"fcf756dc-ea28-4e48-b015-704955375715\") " pod="ai-tenant-e2e-inf-b-fe6ee5/test-model-fe6ee5-kserve-7774b7d84f-rn4bs"
Sep 17 14:15:42.825015 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:42.824856    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcf756dc-ea28-4e48-b015-704955375715-tmp-dir\") pod \"test-model-fe6ee5-kserve-7774b7d84f-rn4bs\" (UID: \"fcf756dc-ea28-4e48-b015-704955375715\") " pod="ai-tenant-e2e-inf-b-fe6ee5/test-model-fe6ee5-kserve-7774b7d84f-rn4bs"
Sep 17 14:15:42.825314 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:42.825293    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcf756dc-ea28-4e48-b015-704955375715-home\") pod \"test-model-fe6ee5-kserve-7774b7d84f-rn4bs\" (UID: \"fcf756dc-ea28-4e48-b015-704955375715\") " pod="ai-tenant-e2e-inf-b-fe6ee5/test-model-fe6ee5-kserve-7774b7d84f-rn4bs"
Sep 17 14:15:42.825719 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:42.825673    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcf756dc-ea28-4e48-b015-704955375715-tmp-dir\") pod \"test-model-fe6ee5-kserve-7774b7d84f-rn4bs\" (UID: \"fcf756dc-ea28-4e48-b015-704955375715\") " pod="ai-tenant-e2e-inf-b-fe6ee5/test-model-fe6ee5-kserve-7774b7d84f-rn4bs"
Sep 17 14:15:42.827579 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:42.826048    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcf756dc-ea28-4e48-b015-704955375715-model-cache\") pod \"test-model-fe6ee5-kserve-7774b7d84f-rn4bs\" (UID: \"fcf756dc-ea28-4e48-b015-704955375715\") " pod="ai-tenant-e2e-inf-b-fe6ee5/test-model-fe6ee5-kserve-7774b7d84f-rn4bs"
Sep 17 14:15:42.829215 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:42.829157    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcf756dc-ea28-4e48-b015-704955375715-dshm\") pod \"test-model-fe6ee5-kserve-7774b7d84f-rn4bs\" (UID: \"fcf756dc-ea28-4e48-b015-704955375715\") " pod="ai-tenant-e2e-inf-b-fe6ee5/test-model-fe6ee5-kserve-7774b7d84f-rn4bs"
Sep 17 14:15:42.829812 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:42.829786    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcf756dc-ea28-4e48-b015-704955375715-tls-certs\") pod \"test-model-fe6ee5-kserve-7774b7d84f-rn4bs\" (UID: \"fcf756dc-ea28-4e48-b015-704955375715\") " pod="ai-tenant-e2e-inf-b-fe6ee5/test-model-fe6ee5-kserve-7774b7d84f-rn4bs"
Sep 17 14:15:42.845884 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:42.845856    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zzf5\" (UniqueName: \"kubernetes.io/projected/fcf756dc-ea28-4e48-b015-704955375715-kube-api-access-6zzf5\") pod \"test-model-fe6ee5-kserve-7774b7d84f-rn4bs\" (UID: \"fcf756dc-ea28-4e48-b015-704955375715\") " pod="ai-tenant-e2e-inf-b-fe6ee5/test-model-fe6ee5-kserve-7774b7d84f-rn4bs"
Sep 17 14:15:43.032294 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:43.032216    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fe6ee5/test-model-fe6ee5-kserve-7774b7d84f-rn4bs"
Sep 17 14:15:43.341248 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:43.341222    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-fe6ee5/test-model-fe6ee5-kserve-7774b7d84f-rn4bs"]
Sep 17 14:15:43.342393 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:15:43.342364    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfcf756dc_ea28_4e48_b015_704955375715.slice/crio-6ad56ba5df3d5c955cdf4f3f367102368b500910dc251a6f454e0c83a62a8113 WatchSource:0}: Error finding container 6ad56ba5df3d5c955cdf4f3f367102368b500910dc251a6f454e0c83a62a8113: Status 404 returned error can't find the container with id 6ad56ba5df3d5c955cdf4f3f367102368b500910dc251a6f454e0c83a62a8113
Sep 17 14:15:44.066599 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:44.066563    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fe6ee5/test-model-fe6ee5-kserve-7774b7d84f-rn4bs" event={"ID":"fcf756dc-ea28-4e48-b015-704955375715","Type":"ContainerStarted","Data":"cb5898f5113991cd53a9e9090624ebda8ba418f501b2a8127b039d2a693f48bb"}
Sep 17 14:15:44.066968 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:44.066607    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fe6ee5/test-model-fe6ee5-kserve-7774b7d84f-rn4bs" event={"ID":"fcf756dc-ea28-4e48-b015-704955375715","Type":"ContainerStarted","Data":"6ad56ba5df3d5c955cdf4f3f367102368b500910dc251a6f454e0c83a62a8113"}
Sep 17 14:15:44.086491 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:44.086432    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-fe6ee5/test-model-fe6ee5-kserve-7774b7d84f-rn4bs" podStartSLOduration=2.086416858 podStartE2EDuration="2.086416858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:15:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:15:44.0844394 +0000 UTC m=+1213.026583046" watchObservedRunningTime="2026-09-17 14:15:44.086416858 +0000 UTC m=+1213.028560551"
Sep 17 14:15:48.404554 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:48.404515    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:15:48.468309 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:48.468280    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:15:49.098149 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:49.098112    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:15:53.032763 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:53.032727    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-fe6ee5/test-model-fe6ee5-kserve-7774b7d84f-rn4bs"
Sep 17 14:15:53.033200 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:53.032781    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-fe6ee5/test-model-fe6ee5-kserve-7774b7d84f-rn4bs"
Sep 17 14:15:53.045439 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:53.045416    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-fe6ee5/test-model-fe6ee5-kserve-7774b7d84f-rn4bs"
Sep 17 14:15:53.134596 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:53.134570    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-fe6ee5/test-model-fe6ee5-kserve-7774b7d84f-rn4bs"
Sep 17 14:15:55.482608 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:55.482574    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:15:55.584326 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:55.584286    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:15:55.765370 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:55.765291    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:15:56.059336 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:56.059309    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x"]
Sep 17 14:15:56.063236 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:56.063220    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x"
Sep 17 14:15:56.065341 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:56.065322    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-4a43447e-openshift-default-dockercfg-6dtgw\""
Sep 17 14:15:56.075472 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:56.075440    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x"]
Sep 17 14:15:56.077003 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:56.076978    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:15:56.142753 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:56.142730    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0097797-0e46-4510-83d6-5755d8a1242b-workload-certs\") pod \"e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x\" (UID: \"f0097797-0e46-4510-83d6-5755d8a1242b\") " pod="openshift-ingress/e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x"
Sep 17 14:15:56.142888 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:56.142765    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4x62\" (UniqueName: \"kubernetes.io/projected/f0097797-0e46-4510-83d6-5755d8a1242b-kube-api-access-p4x62\") pod \"e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x\" (UID: \"f0097797-0e46-4510-83d6-5755d8a1242b\") " pod="openshift-ingress/e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x"
Sep 17 14:15:56.142888 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:56.142827    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0097797-0e46-4510-83d6-5755d8a1242b-istio-token\") pod \"e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x\" (UID: \"f0097797-0e46-4510-83d6-5755d8a1242b\") " pod="openshift-ingress/e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x"
Sep 17 14:15:56.143261 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:56.142898    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0097797-0e46-4510-83d6-5755d8a1242b-istiod-ca-cert\") pod \"e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x\" (UID: \"f0097797-0e46-4510-83d6-5755d8a1242b\") " pod="openshift-ingress/e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x"
Sep 17 14:15:56.143261 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:56.142941    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0097797-0e46-4510-83d6-5755d8a1242b-workload-socket\") pod \"e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x\" (UID: \"f0097797-0e46-4510-83d6-5755d8a1242b\") " pod="openshift-ingress/e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x"
Sep 17 14:15:56.143261 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:56.142976    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0097797-0e46-4510-83d6-5755d8a1242b-istio-envoy\") pod \"e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x\" (UID: \"f0097797-0e46-4510-83d6-5755d8a1242b\") " pod="openshift-ingress/e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x"
Sep 17 14:15:56.143261 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:56.143018    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0097797-0e46-4510-83d6-5755d8a1242b-istio-podinfo\") pod \"e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x\" (UID: \"f0097797-0e46-4510-83d6-5755d8a1242b\") " pod="openshift-ingress/e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x"
Sep 17 14:15:56.143261 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:56.143042    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0097797-0e46-4510-83d6-5755d8a1242b-istio-data\") pod \"e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x\" (UID: \"f0097797-0e46-4510-83d6-5755d8a1242b\") " pod="openshift-ingress/e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x"
Sep 17 14:15:56.143261 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:56.143057    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0097797-0e46-4510-83d6-5755d8a1242b-credential-socket\") pod \"e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x\" (UID: \"f0097797-0e46-4510-83d6-5755d8a1242b\") " pod="openshift-ingress/e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x"
Sep 17 14:15:56.176182 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:56.176156    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:15:56.244563 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:56.244522    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0097797-0e46-4510-83d6-5755d8a1242b-istiod-ca-cert\") pod \"e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x\" (UID: \"f0097797-0e46-4510-83d6-5755d8a1242b\") " pod="openshift-ingress/e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x"
Sep 17 14:15:56.244754 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:56.244628    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0097797-0e46-4510-83d6-5755d8a1242b-workload-socket\") pod \"e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x\" (UID: \"f0097797-0e46-4510-83d6-5755d8a1242b\") " pod="openshift-ingress/e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x"
Sep 17 14:15:56.244754 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:56.244681    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0097797-0e46-4510-83d6-5755d8a1242b-istio-envoy\") pod \"e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x\" (UID: \"f0097797-0e46-4510-83d6-5755d8a1242b\") " pod="openshift-ingress/e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x"
Sep 17 14:15:56.244754 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:56.244732    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0097797-0e46-4510-83d6-5755d8a1242b-istio-podinfo\") pod \"e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x\" (UID: \"f0097797-0e46-4510-83d6-5755d8a1242b\") " pod="openshift-ingress/e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x"
Sep 17 14:15:56.244924 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:56.244761    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0097797-0e46-4510-83d6-5755d8a1242b-istio-data\") pod \"e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x\" (UID: \"f0097797-0e46-4510-83d6-5755d8a1242b\") " pod="openshift-ingress/e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x"
Sep 17 14:15:56.244924 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:56.244794    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0097797-0e46-4510-83d6-5755d8a1242b-credential-socket\") pod \"e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x\" (UID: \"f0097797-0e46-4510-83d6-5755d8a1242b\") " pod="openshift-ingress/e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x"
Sep 17 14:15:56.244924 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:56.244838    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0097797-0e46-4510-83d6-5755d8a1242b-workload-certs\") pod \"e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x\" (UID: \"f0097797-0e46-4510-83d6-5755d8a1242b\") " pod="openshift-ingress/e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x"
Sep 17 14:15:56.244924 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:56.244889    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4x62\" (UniqueName: \"kubernetes.io/projected/f0097797-0e46-4510-83d6-5755d8a1242b-kube-api-access-p4x62\") pod \"e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x\" (UID: \"f0097797-0e46-4510-83d6-5755d8a1242b\") " pod="openshift-ingress/e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x"
Sep 17 14:15:56.245116 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:56.244972    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0097797-0e46-4510-83d6-5755d8a1242b-istio-token\") pod \"e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x\" (UID: \"f0097797-0e46-4510-83d6-5755d8a1242b\") " pod="openshift-ingress/e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x"
Sep 17 14:15:56.246257 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:56.246226    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0097797-0e46-4510-83d6-5755d8a1242b-istiod-ca-cert\") pod \"e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x\" (UID: \"f0097797-0e46-4510-83d6-5755d8a1242b\") " pod="openshift-ingress/e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x"
Sep 17 14:15:56.246582 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:56.246559    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0097797-0e46-4510-83d6-5755d8a1242b-workload-socket\") pod \"e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x\" (UID: \"f0097797-0e46-4510-83d6-5755d8a1242b\") " pod="openshift-ingress/e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x"
Sep 17 14:15:56.251860 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:56.251831    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0097797-0e46-4510-83d6-5755d8a1242b-credential-socket\") pod \"e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x\" (UID: \"f0097797-0e46-4510-83d6-5755d8a1242b\") " pod="openshift-ingress/e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x"
Sep 17 14:15:56.251968 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:56.251885    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0097797-0e46-4510-83d6-5755d8a1242b-workload-certs\") pod \"e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x\" (UID: \"f0097797-0e46-4510-83d6-5755d8a1242b\") " pod="openshift-ingress/e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x"
Sep 17 14:15:56.252035 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:56.252012    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0097797-0e46-4510-83d6-5755d8a1242b-istio-data\") pod \"e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x\" (UID: \"f0097797-0e46-4510-83d6-5755d8a1242b\") " pod="openshift-ingress/e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x"
Sep 17 14:15:56.252671 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:56.252648    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0097797-0e46-4510-83d6-5755d8a1242b-istio-envoy\") pod \"e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x\" (UID: \"f0097797-0e46-4510-83d6-5755d8a1242b\") " pod="openshift-ingress/e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x"
Sep 17 14:15:56.254264 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:56.254242    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0097797-0e46-4510-83d6-5755d8a1242b-istio-podinfo\") pod \"e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x\" (UID: \"f0097797-0e46-4510-83d6-5755d8a1242b\") " pod="openshift-ingress/e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x"
Sep 17 14:15:56.254623 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:56.254605    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0097797-0e46-4510-83d6-5755d8a1242b-istio-token\") pod \"e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x\" (UID: \"f0097797-0e46-4510-83d6-5755d8a1242b\") " pod="openshift-ingress/e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x"
Sep 17 14:15:56.260002 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:56.259974    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4x62\" (UniqueName: \"kubernetes.io/projected/f0097797-0e46-4510-83d6-5755d8a1242b-kube-api-access-p4x62\") pod \"e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x\" (UID: \"f0097797-0e46-4510-83d6-5755d8a1242b\") " pod="openshift-ingress/e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x"
Sep 17 14:15:56.375410 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:56.375351    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x"
Sep 17 14:15:56.710316 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:56.710276    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x"]
Sep 17 14:15:56.711070 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:15:56.711042    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0097797_0e46_4510_83d6_5755d8a1242b.slice/crio-67fa9e26f9bc82ce753d4427020892ab6cfe595ad5db8a11dff0ae85f046c269 WatchSource:0}: Error finding container 67fa9e26f9bc82ce753d4427020892ab6cfe595ad5db8a11dff0ae85f046c269: Status 404 returned error can't find the container with id 67fa9e26f9bc82ce753d4427020892ab6cfe595ad5db8a11dff0ae85f046c269
Sep 17 14:15:56.713619 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:56.713588    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:15:56.713749 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:56.713654    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:15:56.713749 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:56.713707    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:15:57.149247 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:57.149211    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x" event={"ID":"f0097797-0e46-4510-83d6-5755d8a1242b","Type":"ContainerStarted","Data":"23b41f432806776191177c1039058ecf3916f5dcf90268653545afce73e6a59d"}
Sep 17 14:15:57.149413 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:57.149261    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x" event={"ID":"f0097797-0e46-4510-83d6-5755d8a1242b","Type":"ContainerStarted","Data":"67fa9e26f9bc82ce753d4427020892ab6cfe595ad5db8a11dff0ae85f046c269"}
Sep 17 14:15:57.171663 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:57.171617    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x" podStartSLOduration=1.171601182 podStartE2EDuration="1.171601182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:15:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:15:57.168816521 +0000 UTC m=+1226.110960168" watchObservedRunningTime="2026-09-17 14:15:57.171601182 +0000 UTC m=+1226.113744830"
Sep 17 14:15:57.375960 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:57.375926    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x"
Sep 17 14:15:57.377293 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:57.377267    2560 patch_prober.go:28] interesting pod/e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Sep 17 14:15:57.377418 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:57.377313    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x" podUID="f0097797-0e46-4510-83d6-5755d8a1242b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Sep 17 14:15:58.379530 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:58.379505    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x"
Sep 17 14:15:59.161815 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:59.161791    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x"
Sep 17 14:15:59.162453 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:15:59.162437    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x"
Sep 17 14:16:02.253779 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:02.253736    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:02.992121 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:02.992081    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:03.081988 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:03.081949    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:03.331678 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:03.331650    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:04.535901 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:04.535866    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:05.957957 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:05.957901    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:06.668808 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:06.668775    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:07.548950 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:07.548912    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:08.365315 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:08.365282    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:08.441261 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:08.441227    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:08.535895 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:08.535868    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:09.018701 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:09.018667    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:09.117149 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:09.117115    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:21.405631 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:21.405588    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-767170/explicit-model-767170-kserve-9d9f5bc65-kxrd2"]
Sep 17 14:16:21.409597 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:21.409580    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-767170/explicit-model-767170-kserve-9d9f5bc65-kxrd2"
Sep 17 14:16:21.411656 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:21.411632    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-767170\"/\"kube-root-ca.crt\""
Sep 17 14:16:21.411771 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:21.411675    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-767170\"/\"openshift-service-ca.crt\""
Sep 17 14:16:21.412159 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:21.412143    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-767170\"/\"default-dockercfg-flwlf\""
Sep 17 14:16:21.412242 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:21.412215    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-767170\"/\"explicit-model-767170-kserve-self-signed-certs\""
Sep 17 14:16:21.422266 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:21.422245    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-767170/explicit-model-767170-kserve-9d9f5bc65-kxrd2"]
Sep 17 14:16:21.436019 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:21.435995    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:21.597731 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:21.597693    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmc85\" (UniqueName: \"kubernetes.io/projected/085bbbdf-8626-4a8c-be0b-f0ca98d7714d-kube-api-access-qmc85\") pod \"explicit-model-767170-kserve-9d9f5bc65-kxrd2\" (UID: \"085bbbdf-8626-4a8c-be0b-f0ca98d7714d\") " pod="ai-tenant-e2e-explicit-767170/explicit-model-767170-kserve-9d9f5bc65-kxrd2"
Sep 17 14:16:21.597929 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:21.597748    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/085bbbdf-8626-4a8c-be0b-f0ca98d7714d-home\") pod \"explicit-model-767170-kserve-9d9f5bc65-kxrd2\" (UID: \"085bbbdf-8626-4a8c-be0b-f0ca98d7714d\") " pod="ai-tenant-e2e-explicit-767170/explicit-model-767170-kserve-9d9f5bc65-kxrd2"
Sep 17 14:16:21.597929 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:21.597793    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/085bbbdf-8626-4a8c-be0b-f0ca98d7714d-tls-certs\") pod \"explicit-model-767170-kserve-9d9f5bc65-kxrd2\" (UID: \"085bbbdf-8626-4a8c-be0b-f0ca98d7714d\") " pod="ai-tenant-e2e-explicit-767170/explicit-model-767170-kserve-9d9f5bc65-kxrd2"
Sep 17 14:16:21.597929 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:21.597873    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/085bbbdf-8626-4a8c-be0b-f0ca98d7714d-dshm\") pod \"explicit-model-767170-kserve-9d9f5bc65-kxrd2\" (UID: \"085bbbdf-8626-4a8c-be0b-f0ca98d7714d\") " pod="ai-tenant-e2e-explicit-767170/explicit-model-767170-kserve-9d9f5bc65-kxrd2"
Sep 17 14:16:21.598089 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:21.597988    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/085bbbdf-8626-4a8c-be0b-f0ca98d7714d-model-cache\") pod \"explicit-model-767170-kserve-9d9f5bc65-kxrd2\" (UID: \"085bbbdf-8626-4a8c-be0b-f0ca98d7714d\") " pod="ai-tenant-e2e-explicit-767170/explicit-model-767170-kserve-9d9f5bc65-kxrd2"
Sep 17 14:16:21.598145 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:21.598091    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/085bbbdf-8626-4a8c-be0b-f0ca98d7714d-tmp-dir\") pod \"explicit-model-767170-kserve-9d9f5bc65-kxrd2\" (UID: \"085bbbdf-8626-4a8c-be0b-f0ca98d7714d\") " pod="ai-tenant-e2e-explicit-767170/explicit-model-767170-kserve-9d9f5bc65-kxrd2"
Sep 17 14:16:21.698954 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:21.698861    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/085bbbdf-8626-4a8c-be0b-f0ca98d7714d-model-cache\") pod \"explicit-model-767170-kserve-9d9f5bc65-kxrd2\" (UID: \"085bbbdf-8626-4a8c-be0b-f0ca98d7714d\") " pod="ai-tenant-e2e-explicit-767170/explicit-model-767170-kserve-9d9f5bc65-kxrd2"
Sep 17 14:16:21.698954 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:21.698944    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/085bbbdf-8626-4a8c-be0b-f0ca98d7714d-tmp-dir\") pod \"explicit-model-767170-kserve-9d9f5bc65-kxrd2\" (UID: \"085bbbdf-8626-4a8c-be0b-f0ca98d7714d\") " pod="ai-tenant-e2e-explicit-767170/explicit-model-767170-kserve-9d9f5bc65-kxrd2"
Sep 17 14:16:21.699186 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:21.699016    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmc85\" (UniqueName: \"kubernetes.io/projected/085bbbdf-8626-4a8c-be0b-f0ca98d7714d-kube-api-access-qmc85\") pod \"explicit-model-767170-kserve-9d9f5bc65-kxrd2\" (UID: \"085bbbdf-8626-4a8c-be0b-f0ca98d7714d\") " pod="ai-tenant-e2e-explicit-767170/explicit-model-767170-kserve-9d9f5bc65-kxrd2"
Sep 17 14:16:21.699186 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:21.699053    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/085bbbdf-8626-4a8c-be0b-f0ca98d7714d-home\") pod \"explicit-model-767170-kserve-9d9f5bc65-kxrd2\" (UID: \"085bbbdf-8626-4a8c-be0b-f0ca98d7714d\") " pod="ai-tenant-e2e-explicit-767170/explicit-model-767170-kserve-9d9f5bc65-kxrd2"
Sep 17 14:16:21.699186 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:21.699079    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/085bbbdf-8626-4a8c-be0b-f0ca98d7714d-tls-certs\") pod \"explicit-model-767170-kserve-9d9f5bc65-kxrd2\" (UID: \"085bbbdf-8626-4a8c-be0b-f0ca98d7714d\") " pod="ai-tenant-e2e-explicit-767170/explicit-model-767170-kserve-9d9f5bc65-kxrd2"
Sep 17 14:16:21.699186 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:21.699129    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/085bbbdf-8626-4a8c-be0b-f0ca98d7714d-dshm\") pod \"explicit-model-767170-kserve-9d9f5bc65-kxrd2\" (UID: \"085bbbdf-8626-4a8c-be0b-f0ca98d7714d\") " pod="ai-tenant-e2e-explicit-767170/explicit-model-767170-kserve-9d9f5bc65-kxrd2"
Sep 17 14:16:21.699398 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:21.699354    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/085bbbdf-8626-4a8c-be0b-f0ca98d7714d-model-cache\") pod \"explicit-model-767170-kserve-9d9f5bc65-kxrd2\" (UID: \"085bbbdf-8626-4a8c-be0b-f0ca98d7714d\") " pod="ai-tenant-e2e-explicit-767170/explicit-model-767170-kserve-9d9f5bc65-kxrd2"
Sep 17 14:16:21.699506 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:21.699404    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/085bbbdf-8626-4a8c-be0b-f0ca98d7714d-tmp-dir\") pod \"explicit-model-767170-kserve-9d9f5bc65-kxrd2\" (UID: \"085bbbdf-8626-4a8c-be0b-f0ca98d7714d\") " pod="ai-tenant-e2e-explicit-767170/explicit-model-767170-kserve-9d9f5bc65-kxrd2"
Sep 17 14:16:21.699506 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:21.699442    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/085bbbdf-8626-4a8c-be0b-f0ca98d7714d-home\") pod \"explicit-model-767170-kserve-9d9f5bc65-kxrd2\" (UID: \"085bbbdf-8626-4a8c-be0b-f0ca98d7714d\") " pod="ai-tenant-e2e-explicit-767170/explicit-model-767170-kserve-9d9f5bc65-kxrd2"
Sep 17 14:16:21.701447 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:21.701418    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/085bbbdf-8626-4a8c-be0b-f0ca98d7714d-dshm\") pod \"explicit-model-767170-kserve-9d9f5bc65-kxrd2\" (UID: \"085bbbdf-8626-4a8c-be0b-f0ca98d7714d\") " pod="ai-tenant-e2e-explicit-767170/explicit-model-767170-kserve-9d9f5bc65-kxrd2"
Sep 17 14:16:21.701741 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:21.701720    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/085bbbdf-8626-4a8c-be0b-f0ca98d7714d-tls-certs\") pod \"explicit-model-767170-kserve-9d9f5bc65-kxrd2\" (UID: \"085bbbdf-8626-4a8c-be0b-f0ca98d7714d\") " pod="ai-tenant-e2e-explicit-767170/explicit-model-767170-kserve-9d9f5bc65-kxrd2"
Sep 17 14:16:21.709194 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:21.709170    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmc85\" (UniqueName: \"kubernetes.io/projected/085bbbdf-8626-4a8c-be0b-f0ca98d7714d-kube-api-access-qmc85\") pod \"explicit-model-767170-kserve-9d9f5bc65-kxrd2\" (UID: \"085bbbdf-8626-4a8c-be0b-f0ca98d7714d\") " pod="ai-tenant-e2e-explicit-767170/explicit-model-767170-kserve-9d9f5bc65-kxrd2"
Sep 17 14:16:21.720583 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:21.720562    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-767170/explicit-model-767170-kserve-9d9f5bc65-kxrd2"
Sep 17 14:16:22.023924 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:22.023808    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-767170/explicit-model-767170-kserve-9d9f5bc65-kxrd2"]
Sep 17 14:16:22.029873 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:16:22.029834    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod085bbbdf_8626_4a8c_be0b_f0ca98d7714d.slice/crio-d1884e0cd9bed2b665c388d2a91098f8c1b11b08bd6a2b1bbbb7e12037d04579 WatchSource:0}: Error finding container d1884e0cd9bed2b665c388d2a91098f8c1b11b08bd6a2b1bbbb7e12037d04579: Status 404 returned error can't find the container with id d1884e0cd9bed2b665c388d2a91098f8c1b11b08bd6a2b1bbbb7e12037d04579
Sep 17 14:16:22.305484 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:22.305424    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-767170/explicit-model-767170-kserve-9d9f5bc65-kxrd2" event={"ID":"085bbbdf-8626-4a8c-be0b-f0ca98d7714d","Type":"ContainerStarted","Data":"118222f4f1102823f4ddeab20aaeef717218e81feeb5538b1c1d365cee60d6d2"}
Sep 17 14:16:22.305661 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:22.305498    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-767170/explicit-model-767170-kserve-9d9f5bc65-kxrd2" event={"ID":"085bbbdf-8626-4a8c-be0b-f0ca98d7714d","Type":"ContainerStarted","Data":"d1884e0cd9bed2b665c388d2a91098f8c1b11b08bd6a2b1bbbb7e12037d04579"}
Sep 17 14:16:22.337144 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:22.337097    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-767170/explicit-model-767170-kserve-9d9f5bc65-kxrd2" podStartSLOduration=1.337077758 podStartE2EDuration="1.337077758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:16:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:16:22.335777104 +0000 UTC m=+1251.277920741" watchObservedRunningTime="2026-09-17 14:16:22.337077758 +0000 UTC m=+1251.279221426"
Sep 17 14:16:25.263115 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:25.263016    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:27.846041 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:27.845949    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:30.202652 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:30.202609    2560 patch_prober.go:28] interesting pod/e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 17 14:16:30.203140 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:30.202684    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x" podUID="f0097797-0e46-4510-83d6-5755d8a1242b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 17 14:16:31.068696 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:31.068663    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:31.368579 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:31.368496    2560 generic.go:356] "Generic (PLEG): container finished" podID="f0097797-0e46-4510-83d6-5755d8a1242b" containerID="23b41f432806776191177c1039058ecf3916f5dcf90268653545afce73e6a59d" exitCode=0
Sep 17 14:16:31.368925 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:31.368570    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x" event={"ID":"f0097797-0e46-4510-83d6-5755d8a1242b","Type":"ContainerDied","Data":"23b41f432806776191177c1039058ecf3916f5dcf90268653545afce73e6a59d"}
Sep 17 14:16:31.368976 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:31.368939    2560 scope.go:117] "RemoveContainer" containerID="23b41f432806776191177c1039058ecf3916f5dcf90268653545afce73e6a59d"
Sep 17 14:16:31.370080 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:31.370055    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:16:31.370140 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:31.370119    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:16:31.370177 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:31.370148    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:16:31.721143 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:31.721067    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-767170/explicit-model-767170-kserve-9d9f5bc65-kxrd2"
Sep 17 14:16:31.721143 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:31.721117    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-767170/explicit-model-767170-kserve-9d9f5bc65-kxrd2"
Sep 17 14:16:31.734218 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:31.734196    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-767170/explicit-model-767170-kserve-9d9f5bc65-kxrd2"
Sep 17 14:16:31.888309 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:31.888277    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:32.389160 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:32.389125    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x" event={"ID":"f0097797-0e46-4510-83d6-5755d8a1242b","Type":"ContainerStarted","Data":"e6fd7f3dd83c5a13915e00a0a1598597de428bba365c8b472cbdacfe49a2d789"}
Sep 17 14:16:32.400432 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:32.400406    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-767170/explicit-model-767170-kserve-9d9f5bc65-kxrd2"
Sep 17 14:16:32.782261 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:32.782168    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:33.049819 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:33.049785    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:33.376666 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:33.376575    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x"
Sep 17 14:16:33.376952 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:33.376821    2560 patch_prober.go:28] interesting pod/e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Sep 17 14:16:33.377097 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:33.377074    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x" podUID="f0097797-0e46-4510-83d6-5755d8a1242b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Sep 17 14:16:33.823426 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:33.823393    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:33.925217 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:33.925158    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:34.298016 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:34.297925    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:34.376261 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:34.376213    2560 patch_prober.go:28] interesting pod/e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Sep 17 14:16:34.376473 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:34.376292    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x" podUID="f0097797-0e46-4510-83d6-5755d8a1242b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Sep 17 14:16:34.972338 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:34.972304    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:35.376138 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:35.376101    2560 patch_prober.go:28] interesting pod/e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Sep 17 14:16:35.376441 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:35.376410    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x" podUID="f0097797-0e46-4510-83d6-5755d8a1242b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Sep 17 14:16:35.815933 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:35.815903    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:36.375765 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:36.375731    2560 patch_prober.go:28] interesting pod/e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Sep 17 14:16:36.376319 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:36.376284    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x" podUID="f0097797-0e46-4510-83d6-5755d8a1242b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Sep 17 14:16:36.661258 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:36.661162    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:37.375937 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:37.375896    2560 patch_prober.go:28] interesting pod/e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Sep 17 14:16:37.376306 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:37.375959    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x" podUID="f0097797-0e46-4510-83d6-5755d8a1242b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Sep 17 14:16:38.379780 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:38.379751    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x"
Sep 17 14:16:38.380211 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:38.380047    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x"
Sep 17 14:16:38.380786 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:38.380769    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x"
Sep 17 14:16:40.742448 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:40.742414    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:40.844312 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:40.844280    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:41.151660 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:41.151625    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a43447e-rlgqq"]
Sep 17 14:16:41.155711 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:41.155690    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a43447e-rlgqq"
Sep 17 14:16:41.158010 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:41.157978    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 14:16:41.158138 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:41.158122    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rp9ww\""
Sep 17 14:16:41.162875 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:41.162855    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a43447e-rlgqq"]
Sep 17 14:16:41.210316 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:41.210288    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17a1c74b-939e-474b-b238-abfeada3f8e5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-4a43447e-rlgqq\" (UID: \"17a1c74b-939e-474b-b238-abfeada3f8e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a43447e-rlgqq"
Sep 17 14:16:41.311732 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:41.311702    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17a1c74b-939e-474b-b238-abfeada3f8e5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-4a43447e-rlgqq\" (UID: \"17a1c74b-939e-474b-b238-abfeada3f8e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a43447e-rlgqq"
Sep 17 14:16:41.312378 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:41.312357    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17a1c74b-939e-474b-b238-abfeada3f8e5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-4a43447e-rlgqq\" (UID: \"17a1c74b-939e-474b-b238-abfeada3f8e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a43447e-rlgqq"
Sep 17 14:16:41.467949 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:41.467873    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a43447e-rlgqq"
Sep 17 14:16:41.749888 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:41.749862    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a43447e-rlgqq"]
Sep 17 14:16:41.751026 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:16:41.751004    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17a1c74b_939e_474b_b238_abfeada3f8e5.slice/crio-1990fc781e61a3813892c1f9dd0ebd5a6d82139f3b0bcb567ad8ad2cd2d5e834 WatchSource:0}: Error finding container 1990fc781e61a3813892c1f9dd0ebd5a6d82139f3b0bcb567ad8ad2cd2d5e834: Status 404 returned error can't find the container with id 1990fc781e61a3813892c1f9dd0ebd5a6d82139f3b0bcb567ad8ad2cd2d5e834
Sep 17 14:16:42.178070 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:42.178038    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:42.455726 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:42.455638    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a43447e-rlgqq" event={"ID":"17a1c74b-939e-474b-b238-abfeada3f8e5","Type":"ContainerStarted","Data":"47d1d9fd19cc65bfc5a1a1f6b4c6489b8ca534819be28ad3a6abb1db01586d60"}
Sep 17 14:16:42.455726 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:42.455698    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a43447e-rlgqq" event={"ID":"17a1c74b-939e-474b-b238-abfeada3f8e5","Type":"ContainerStarted","Data":"1990fc781e61a3813892c1f9dd0ebd5a6d82139f3b0bcb567ad8ad2cd2d5e834"}
Sep 17 14:16:42.475345 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:42.475291    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a43447e-rlgqq" podStartSLOduration=1.475276638 podStartE2EDuration="1.475276638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:16:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:16:42.471836036 +0000 UTC m=+1271.413979683" watchObservedRunningTime="2026-09-17 14:16:42.475276638 +0000 UTC m=+1271.417420285"
Sep 17 14:16:42.958802 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:42.958770    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:43.463578 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:43.463542    2560 generic.go:356] "Generic (PLEG): container finished" podID="17a1c74b-939e-474b-b238-abfeada3f8e5" containerID="47d1d9fd19cc65bfc5a1a1f6b4c6489b8ca534819be28ad3a6abb1db01586d60" exitCode=0
Sep 17 14:16:43.463753 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:43.463611    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a43447e-rlgqq" event={"ID":"17a1c74b-939e-474b-b238-abfeada3f8e5","Type":"ContainerDied","Data":"47d1d9fd19cc65bfc5a1a1f6b4c6489b8ca534819be28ad3a6abb1db01586d60"}
Sep 17 14:16:43.785083 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:43.784996    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:44.602999 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:44.602974    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a43447e-rlgqq"
Sep 17 14:16:44.751394 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:44.751304    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17a1c74b-939e-474b-b238-abfeada3f8e5-maas-api-service-ca\") pod \"17a1c74b-939e-474b-b238-abfeada3f8e5\" (UID: \"17a1c74b-939e-474b-b238-abfeada3f8e5\") "
Sep 17 14:16:44.751689 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:44.751668    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17a1c74b-939e-474b-b238-abfeada3f8e5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "17a1c74b-939e-474b-b238-abfeada3f8e5" (UID: "17a1c74b-939e-474b-b238-abfeada3f8e5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:16:44.777382 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:44.777346    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:44.852979 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:44.852945    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17a1c74b-939e-474b-b238-abfeada3f8e5-maas-api-service-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:16:45.478626 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:45.478598    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a43447e-rlgqq"
Sep 17 14:16:45.478879 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:45.478591    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a43447e-rlgqq" event={"ID":"17a1c74b-939e-474b-b238-abfeada3f8e5","Type":"ContainerDied","Data":"1990fc781e61a3813892c1f9dd0ebd5a6d82139f3b0bcb567ad8ad2cd2d5e834"}
Sep 17 14:16:45.478879 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:45.478719    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1990fc781e61a3813892c1f9dd0ebd5a6d82139f3b0bcb567ad8ad2cd2d5e834"
Sep 17 14:16:45.777975 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:45.777896    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:46.654516 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:46.654485    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:47.478099 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:47.478061    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:49.529437 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:49.529403    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-d9b17d/distinct-w0-d9b17d-kserve-85bc8c598b-cflt6"]
Sep 17 14:16:49.529903 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:49.529862    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17a1c74b-939e-474b-b238-abfeada3f8e5" containerName="revoke-keys"
Sep 17 14:16:49.529903 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:49.529874    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="17a1c74b-939e-474b-b238-abfeada3f8e5" containerName="revoke-keys"
Sep 17 14:16:49.529978 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:49.529961    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="17a1c74b-939e-474b-b238-abfeada3f8e5" containerName="revoke-keys"
Sep 17 14:16:49.533082 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:49.533065    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d9b17d/distinct-w0-d9b17d-kserve-85bc8c598b-cflt6"
Sep 17 14:16:49.535237 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:49.535218    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d9b17d\"/\"distinct-w0-d9b17d-kserve-self-signed-certs\""
Sep 17 14:16:49.543472 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:49.543439    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d9b17d/distinct-w0-d9b17d-kserve-85bc8c598b-cflt6"]
Sep 17 14:16:49.647857 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:49.647815    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:49.700473 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:49.700443    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c15de4b-4e65-4d7e-8155-55d3d9241e7b-dshm\") pod \"distinct-w0-d9b17d-kserve-85bc8c598b-cflt6\" (UID: \"2c15de4b-4e65-4d7e-8155-55d3d9241e7b\") " pod="e2e-models-e2e-worker-w0-d9b17d/distinct-w0-d9b17d-kserve-85bc8c598b-cflt6"
Sep 17 14:16:49.700623 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:49.700518    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c15de4b-4e65-4d7e-8155-55d3d9241e7b-model-cache\") pod \"distinct-w0-d9b17d-kserve-85bc8c598b-cflt6\" (UID: \"2c15de4b-4e65-4d7e-8155-55d3d9241e7b\") " pod="e2e-models-e2e-worker-w0-d9b17d/distinct-w0-d9b17d-kserve-85bc8c598b-cflt6"
Sep 17 14:16:49.700623 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:49.700555    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c15de4b-4e65-4d7e-8155-55d3d9241e7b-home\") pod \"distinct-w0-d9b17d-kserve-85bc8c598b-cflt6\" (UID: \"2c15de4b-4e65-4d7e-8155-55d3d9241e7b\") " pod="e2e-models-e2e-worker-w0-d9b17d/distinct-w0-d9b17d-kserve-85bc8c598b-cflt6"
Sep 17 14:16:49.700623 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:49.700614    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c15de4b-4e65-4d7e-8155-55d3d9241e7b-tls-certs\") pod \"distinct-w0-d9b17d-kserve-85bc8c598b-cflt6\" (UID: \"2c15de4b-4e65-4d7e-8155-55d3d9241e7b\") " pod="e2e-models-e2e-worker-w0-d9b17d/distinct-w0-d9b17d-kserve-85bc8c598b-cflt6"
Sep 17 14:16:49.700787 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:49.700748    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpv5t\" (UniqueName: \"kubernetes.io/projected/2c15de4b-4e65-4d7e-8155-55d3d9241e7b-kube-api-access-lpv5t\") pod \"distinct-w0-d9b17d-kserve-85bc8c598b-cflt6\" (UID: \"2c15de4b-4e65-4d7e-8155-55d3d9241e7b\") " pod="e2e-models-e2e-worker-w0-d9b17d/distinct-w0-d9b17d-kserve-85bc8c598b-cflt6"
Sep 17 14:16:49.700834 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:49.700787    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c15de4b-4e65-4d7e-8155-55d3d9241e7b-tmp-dir\") pod \"distinct-w0-d9b17d-kserve-85bc8c598b-cflt6\" (UID: \"2c15de4b-4e65-4d7e-8155-55d3d9241e7b\") " pod="e2e-models-e2e-worker-w0-d9b17d/distinct-w0-d9b17d-kserve-85bc8c598b-cflt6"
Sep 17 14:16:49.802115 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:49.802080    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c15de4b-4e65-4d7e-8155-55d3d9241e7b-dshm\") pod \"distinct-w0-d9b17d-kserve-85bc8c598b-cflt6\" (UID: \"2c15de4b-4e65-4d7e-8155-55d3d9241e7b\") " pod="e2e-models-e2e-worker-w0-d9b17d/distinct-w0-d9b17d-kserve-85bc8c598b-cflt6"
Sep 17 14:16:49.802280 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:49.802145    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c15de4b-4e65-4d7e-8155-55d3d9241e7b-model-cache\") pod \"distinct-w0-d9b17d-kserve-85bc8c598b-cflt6\" (UID: \"2c15de4b-4e65-4d7e-8155-55d3d9241e7b\") " pod="e2e-models-e2e-worker-w0-d9b17d/distinct-w0-d9b17d-kserve-85bc8c598b-cflt6"
Sep 17 14:16:49.802280 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:49.802179    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c15de4b-4e65-4d7e-8155-55d3d9241e7b-home\") pod \"distinct-w0-d9b17d-kserve-85bc8c598b-cflt6\" (UID: \"2c15de4b-4e65-4d7e-8155-55d3d9241e7b\") " pod="e2e-models-e2e-worker-w0-d9b17d/distinct-w0-d9b17d-kserve-85bc8c598b-cflt6"
Sep 17 14:16:49.802280 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:49.802213    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c15de4b-4e65-4d7e-8155-55d3d9241e7b-tls-certs\") pod \"distinct-w0-d9b17d-kserve-85bc8c598b-cflt6\" (UID: \"2c15de4b-4e65-4d7e-8155-55d3d9241e7b\") " pod="e2e-models-e2e-worker-w0-d9b17d/distinct-w0-d9b17d-kserve-85bc8c598b-cflt6"
Sep 17 14:16:49.802454 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:49.802347    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpv5t\" (UniqueName: \"kubernetes.io/projected/2c15de4b-4e65-4d7e-8155-55d3d9241e7b-kube-api-access-lpv5t\") pod \"distinct-w0-d9b17d-kserve-85bc8c598b-cflt6\" (UID: \"2c15de4b-4e65-4d7e-8155-55d3d9241e7b\") " pod="e2e-models-e2e-worker-w0-d9b17d/distinct-w0-d9b17d-kserve-85bc8c598b-cflt6"
Sep 17 14:16:49.802454 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:49.802391    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c15de4b-4e65-4d7e-8155-55d3d9241e7b-tmp-dir\") pod \"distinct-w0-d9b17d-kserve-85bc8c598b-cflt6\" (UID: \"2c15de4b-4e65-4d7e-8155-55d3d9241e7b\") " pod="e2e-models-e2e-worker-w0-d9b17d/distinct-w0-d9b17d-kserve-85bc8c598b-cflt6"
Sep 17 14:16:49.802640 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:49.802621    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c15de4b-4e65-4d7e-8155-55d3d9241e7b-model-cache\") pod \"distinct-w0-d9b17d-kserve-85bc8c598b-cflt6\" (UID: \"2c15de4b-4e65-4d7e-8155-55d3d9241e7b\") " pod="e2e-models-e2e-worker-w0-d9b17d/distinct-w0-d9b17d-kserve-85bc8c598b-cflt6"
Sep 17 14:16:49.802709 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:49.802657    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c15de4b-4e65-4d7e-8155-55d3d9241e7b-home\") pod \"distinct-w0-d9b17d-kserve-85bc8c598b-cflt6\" (UID: \"2c15de4b-4e65-4d7e-8155-55d3d9241e7b\") " pod="e2e-models-e2e-worker-w0-d9b17d/distinct-w0-d9b17d-kserve-85bc8c598b-cflt6"
Sep 17 14:16:49.802775 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:49.802759    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c15de4b-4e65-4d7e-8155-55d3d9241e7b-tmp-dir\") pod \"distinct-w0-d9b17d-kserve-85bc8c598b-cflt6\" (UID: \"2c15de4b-4e65-4d7e-8155-55d3d9241e7b\") " pod="e2e-models-e2e-worker-w0-d9b17d/distinct-w0-d9b17d-kserve-85bc8c598b-cflt6"
Sep 17 14:16:49.804330 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:49.804306    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c15de4b-4e65-4d7e-8155-55d3d9241e7b-dshm\") pod \"distinct-w0-d9b17d-kserve-85bc8c598b-cflt6\" (UID: \"2c15de4b-4e65-4d7e-8155-55d3d9241e7b\") " pod="e2e-models-e2e-worker-w0-d9b17d/distinct-w0-d9b17d-kserve-85bc8c598b-cflt6"
Sep 17 14:16:49.804771 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:49.804753    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c15de4b-4e65-4d7e-8155-55d3d9241e7b-tls-certs\") pod \"distinct-w0-d9b17d-kserve-85bc8c598b-cflt6\" (UID: \"2c15de4b-4e65-4d7e-8155-55d3d9241e7b\") " pod="e2e-models-e2e-worker-w0-d9b17d/distinct-w0-d9b17d-kserve-85bc8c598b-cflt6"
Sep 17 14:16:49.811008 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:49.810987    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpv5t\" (UniqueName: \"kubernetes.io/projected/2c15de4b-4e65-4d7e-8155-55d3d9241e7b-kube-api-access-lpv5t\") pod \"distinct-w0-d9b17d-kserve-85bc8c598b-cflt6\" (UID: \"2c15de4b-4e65-4d7e-8155-55d3d9241e7b\") " pod="e2e-models-e2e-worker-w0-d9b17d/distinct-w0-d9b17d-kserve-85bc8c598b-cflt6"
Sep 17 14:16:49.843673 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:49.843651    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d9b17d/distinct-w0-d9b17d-kserve-85bc8c598b-cflt6"
Sep 17 14:16:50.124333 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:16:50.124306    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2c15de4b_4e65_4d7e_8155_55d3d9241e7b.slice/crio-0bc66c58f6b714b9c9bfd5d6765becb9759a4ece624737e53d81a08d42b24c35 WatchSource:0}: Error finding container 0bc66c58f6b714b9c9bfd5d6765becb9759a4ece624737e53d81a08d42b24c35: Status 404 returned error can't find the container with id 0bc66c58f6b714b9c9bfd5d6765becb9759a4ece624737e53d81a08d42b24c35
Sep 17 14:16:50.124984 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:50.124962    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d9b17d/distinct-w0-d9b17d-kserve-85bc8c598b-cflt6"]
Sep 17 14:16:50.511204 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:50.511123    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d9b17d/distinct-w0-d9b17d-kserve-85bc8c598b-cflt6" event={"ID":"2c15de4b-4e65-4d7e-8155-55d3d9241e7b","Type":"ContainerStarted","Data":"4d8971137530bc706fe961351bddda9f056790b5520bca8f1f6db28218e46d47"}
Sep 17 14:16:50.511204 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:50.511168    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d9b17d/distinct-w0-d9b17d-kserve-85bc8c598b-cflt6" event={"ID":"2c15de4b-4e65-4d7e-8155-55d3d9241e7b","Type":"ContainerStarted","Data":"0bc66c58f6b714b9c9bfd5d6765becb9759a4ece624737e53d81a08d42b24c35"}
Sep 17 14:16:50.531211 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:50.531153    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-d9b17d/distinct-w0-d9b17d-kserve-85bc8c598b-cflt6" podStartSLOduration=1.531137542 podStartE2EDuration="1.531137542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:16:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:16:50.528605795 +0000 UTC m=+1279.470749445" watchObservedRunningTime="2026-09-17 14:16:50.531137542 +0000 UTC m=+1279.473281210"
Sep 17 14:16:52.271506 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:52.271366    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:52.350724 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:52.350686    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a43447e-rlgqq"]
Sep 17 14:16:52.367138 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:52.367101    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a43447e-rlgqq"]
Sep 17 14:16:52.389054 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:52.389024    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:52.685450 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:52.685417    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-767170-8grn5"]
Sep 17 14:16:52.689307 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:52.689286    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-767170-8grn5"
Sep 17 14:16:52.691714 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:52.691683    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 14:16:52.691848 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:52.691833    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rp9ww\""
Sep 17 14:16:52.693742 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:52.693706    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:52.703356 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:52.703336    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-767170-8grn5"]
Sep 17 14:16:52.786977 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:52.786948    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:52.834709 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:52.834676    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cbbbe5a-81dc-4bfa-9dd0-50eb76474f84-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-767170-8grn5\" (UID: \"8cbbbe5a-81dc-4bfa-9dd0-50eb76474f84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-767170-8grn5"
Sep 17 14:16:52.935795 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:52.935719    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cbbbe5a-81dc-4bfa-9dd0-50eb76474f84-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-767170-8grn5\" (UID: \"8cbbbe5a-81dc-4bfa-9dd0-50eb76474f84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-767170-8grn5"
Sep 17 14:16:52.936367 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:52.936342    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cbbbe5a-81dc-4bfa-9dd0-50eb76474f84-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-767170-8grn5\" (UID: \"8cbbbe5a-81dc-4bfa-9dd0-50eb76474f84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-767170-8grn5"
Sep 17 14:16:53.000860 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:53.000837    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-767170-8grn5"
Sep 17 14:16:53.276054 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:53.276022    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-767170-8grn5"]
Sep 17 14:16:53.276702 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:16:53.276676    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cbbbe5a_81dc_4bfa_9dd0_50eb76474f84.slice/crio-858861db456de3bea1b38bdcb6cd6db8394e1d986843d96ad5d6b610e9f606df WatchSource:0}: Error finding container 858861db456de3bea1b38bdcb6cd6db8394e1d986843d96ad5d6b610e9f606df: Status 404 returned error can't find the container with id 858861db456de3bea1b38bdcb6cd6db8394e1d986843d96ad5d6b610e9f606df
Sep 17 14:16:53.531063 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:53.530990    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-767170-8grn5" event={"ID":"8cbbbe5a-81dc-4bfa-9dd0-50eb76474f84","Type":"ContainerStarted","Data":"858861db456de3bea1b38bdcb6cd6db8394e1d986843d96ad5d6b610e9f606df"}
Sep 17 14:16:53.571042 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:53.571020    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17a1c74b-939e-474b-b238-abfeada3f8e5" path="/var/lib/kubelet/pods/17a1c74b-939e-474b-b238-abfeada3f8e5/volumes"
Sep 17 14:16:53.837711 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:53.837685    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:54.539218 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:54.539189    2560 generic.go:356] "Generic (PLEG): container finished" podID="8cbbbe5a-81dc-4bfa-9dd0-50eb76474f84" containerID="87d7fc26f3de138e9850b59c88efac3684b674305f87a492199005aed3b9f8ac" exitCode=0
Sep 17 14:16:54.539616 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:54.539279    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-767170-8grn5" event={"ID":"8cbbbe5a-81dc-4bfa-9dd0-50eb76474f84","Type":"ContainerDied","Data":"87d7fc26f3de138e9850b59c88efac3684b674305f87a492199005aed3b9f8ac"}
Sep 17 14:16:54.674917 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:54.674884    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:55.114083 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:55.114045    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-767170/explicit-model-767170-kserve-9d9f5bc65-kxrd2"]
Sep 17 14:16:55.122158 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:55.122127    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:55.592724 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:55.592691    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:55.686188 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:55.686166    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-767170-8grn5"
Sep 17 14:16:55.864059 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:55.864029    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cbbbe5a-81dc-4bfa-9dd0-50eb76474f84-maas-api-service-ca\") pod \"8cbbbe5a-81dc-4bfa-9dd0-50eb76474f84\" (UID: \"8cbbbe5a-81dc-4bfa-9dd0-50eb76474f84\") "
Sep 17 14:16:55.864369 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:55.864348    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cbbbe5a-81dc-4bfa-9dd0-50eb76474f84-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8cbbbe5a-81dc-4bfa-9dd0-50eb76474f84" (UID: "8cbbbe5a-81dc-4bfa-9dd0-50eb76474f84"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:16:55.964870 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:55.964847    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cbbbe5a-81dc-4bfa-9dd0-50eb76474f84-maas-api-service-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:16:56.554580 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:56.554550    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-767170-8grn5"
Sep 17 14:16:56.554751 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:56.554588    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-767170-8grn5" event={"ID":"8cbbbe5a-81dc-4bfa-9dd0-50eb76474f84","Type":"ContainerDied","Data":"858861db456de3bea1b38bdcb6cd6db8394e1d986843d96ad5d6b610e9f606df"}
Sep 17 14:16:56.554751 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:56.554632    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="858861db456de3bea1b38bdcb6cd6db8394e1d986843d96ad5d6b610e9f606df"
Sep 17 14:16:56.632452 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:56.632423    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:57.623633 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:57.623595    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:58.647013 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:58.646970    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:59.699641 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:59.699607    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:59.844036 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:59.844008    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-d9b17d/distinct-w0-d9b17d-kserve-85bc8c598b-cflt6"
Sep 17 14:16:59.844218 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:59.844057    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d9b17d/distinct-w0-d9b17d-kserve-85bc8c598b-cflt6"
Sep 17 14:16:59.856636 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:16:59.856612    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-d9b17d/distinct-w0-d9b17d-kserve-85bc8c598b-cflt6"
Sep 17 14:17:00.593362 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:00.593336    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d9b17d/distinct-w0-d9b17d-kserve-85bc8c598b-cflt6"
Sep 17 14:17:01.224733 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:01.224697    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:02.185872 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:02.185833    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:02.314209 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:02.314172    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:02.419591 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:02.419549    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:02.434993 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:02.434968    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x"]
Sep 17 14:17:02.435212 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:02.435191    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x" podUID="f0097797-0e46-4510-83d6-5755d8a1242b" containerName="istio-proxy" containerID="cri-o://e6fd7f3dd83c5a13915e00a0a1598597de428bba365c8b472cbdacfe49a2d789" gracePeriod=30
Sep 17 14:17:02.516570 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:02.516477    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:02.850743 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:02.850711    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:02.976043 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:02.976007    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:03.083133 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:03.083084    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:03.441248 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:03.441207    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:03.700578 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:03.700481    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:03.805401 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:03.805366    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:03.950115 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:03.949306    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:03.981230 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:03.981149    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-767170-8grn5"]
Sep 17 14:17:03.984794 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:03.984769    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-767170-8grn5"]
Sep 17 14:17:04.032978 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:04.032946    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:04.403050 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:04.403021    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:04.524024 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:04.523998    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:05.570131 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:05.570097    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8cbbbe5a-81dc-4bfa-9dd0-50eb76474f84" path="/var/lib/kubelet/pods/8cbbbe5a-81dc-4bfa-9dd0-50eb76474f84/volumes"
Sep 17 14:17:05.590356 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:05.590321    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:06.449679 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:06.449640    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:07.241230 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:07.241199    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:07.631875 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:07.631842    2560 generic.go:356] "Generic (PLEG): container finished" podID="f0097797-0e46-4510-83d6-5755d8a1242b" containerID="e6fd7f3dd83c5a13915e00a0a1598597de428bba365c8b472cbdacfe49a2d789" exitCode=0
Sep 17 14:17:07.631875 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:07.631868    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x" event={"ID":"f0097797-0e46-4510-83d6-5755d8a1242b","Type":"ContainerDied","Data":"e6fd7f3dd83c5a13915e00a0a1598597de428bba365c8b472cbdacfe49a2d789"}
Sep 17 14:17:07.632079 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:07.631929    2560 scope.go:117] "RemoveContainer" containerID="23b41f432806776191177c1039058ecf3916f5dcf90268653545afce73e6a59d"
Sep 17 14:17:07.695263 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:07.695240    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x"
Sep 17 14:17:07.793976 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:07.793947    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0097797-0e46-4510-83d6-5755d8a1242b-istio-data\") pod \"f0097797-0e46-4510-83d6-5755d8a1242b\" (UID: \"f0097797-0e46-4510-83d6-5755d8a1242b\") "
Sep 17 14:17:07.794112 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:07.793985    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0097797-0e46-4510-83d6-5755d8a1242b-istio-envoy\") pod \"f0097797-0e46-4510-83d6-5755d8a1242b\" (UID: \"f0097797-0e46-4510-83d6-5755d8a1242b\") "
Sep 17 14:17:07.794112 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:07.794024    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0097797-0e46-4510-83d6-5755d8a1242b-istiod-ca-cert\") pod \"f0097797-0e46-4510-83d6-5755d8a1242b\" (UID: \"f0097797-0e46-4510-83d6-5755d8a1242b\") "
Sep 17 14:17:07.794112 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:07.794067    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0097797-0e46-4510-83d6-5755d8a1242b-istio-podinfo\") pod \"f0097797-0e46-4510-83d6-5755d8a1242b\" (UID: \"f0097797-0e46-4510-83d6-5755d8a1242b\") "
Sep 17 14:17:07.794112 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:07.794107    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p4x62\" (UniqueName: \"kubernetes.io/projected/f0097797-0e46-4510-83d6-5755d8a1242b-kube-api-access-p4x62\") pod \"f0097797-0e46-4510-83d6-5755d8a1242b\" (UID: \"f0097797-0e46-4510-83d6-5755d8a1242b\") "
Sep 17 14:17:07.794277 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:07.794234    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0097797-0e46-4510-83d6-5755d8a1242b-credential-socket\") pod \"f0097797-0e46-4510-83d6-5755d8a1242b\" (UID: \"f0097797-0e46-4510-83d6-5755d8a1242b\") "
Sep 17 14:17:07.794277 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:07.794261    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0097797-0e46-4510-83d6-5755d8a1242b-istio-data" (OuterVolumeSpecName: "istio-data") pod "f0097797-0e46-4510-83d6-5755d8a1242b" (UID: "f0097797-0e46-4510-83d6-5755d8a1242b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:17:07.794380 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:07.794312    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0097797-0e46-4510-83d6-5755d8a1242b-workload-certs\") pod \"f0097797-0e46-4510-83d6-5755d8a1242b\" (UID: \"f0097797-0e46-4510-83d6-5755d8a1242b\") "
Sep 17 14:17:07.794380 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:07.794342    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0097797-0e46-4510-83d6-5755d8a1242b-istio-token\") pod \"f0097797-0e46-4510-83d6-5755d8a1242b\" (UID: \"f0097797-0e46-4510-83d6-5755d8a1242b\") "
Sep 17 14:17:07.794380 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:07.794367    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0097797-0e46-4510-83d6-5755d8a1242b-workload-socket\") pod \"f0097797-0e46-4510-83d6-5755d8a1242b\" (UID: \"f0097797-0e46-4510-83d6-5755d8a1242b\") "
Sep 17 14:17:07.794641 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:07.794498    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0097797-0e46-4510-83d6-5755d8a1242b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f0097797-0e46-4510-83d6-5755d8a1242b" (UID: "f0097797-0e46-4510-83d6-5755d8a1242b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:17:07.794641 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:07.794577    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0097797-0e46-4510-83d6-5755d8a1242b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f0097797-0e46-4510-83d6-5755d8a1242b" (UID: "f0097797-0e46-4510-83d6-5755d8a1242b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:17:07.794641 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:07.794593    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0097797-0e46-4510-83d6-5755d8a1242b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f0097797-0e46-4510-83d6-5755d8a1242b" (UID: "f0097797-0e46-4510-83d6-5755d8a1242b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:17:07.794875 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:07.794777    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0097797-0e46-4510-83d6-5755d8a1242b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f0097797-0e46-4510-83d6-5755d8a1242b" (UID: "f0097797-0e46-4510-83d6-5755d8a1242b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:17:07.794938 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:07.794896    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0097797-0e46-4510-83d6-5755d8a1242b-credential-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:17:07.794938 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:07.794915    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0097797-0e46-4510-83d6-5755d8a1242b-workload-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:17:07.794938 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:07.794930    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0097797-0e46-4510-83d6-5755d8a1242b-workload-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:17:07.795070 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:07.794943    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0097797-0e46-4510-83d6-5755d8a1242b-istio-data\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:17:07.795070 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:07.794957    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0097797-0e46-4510-83d6-5755d8a1242b-istiod-ca-cert\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:17:07.796118 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:07.796095    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0097797-0e46-4510-83d6-5755d8a1242b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f0097797-0e46-4510-83d6-5755d8a1242b" (UID: "f0097797-0e46-4510-83d6-5755d8a1242b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:17:07.796437 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:07.796417    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0097797-0e46-4510-83d6-5755d8a1242b-istio-token" (OuterVolumeSpecName: "istio-token") pod "f0097797-0e46-4510-83d6-5755d8a1242b" (UID: "f0097797-0e46-4510-83d6-5755d8a1242b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:17:07.796637 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:07.796619    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0097797-0e46-4510-83d6-5755d8a1242b-kube-api-access-p4x62" (OuterVolumeSpecName: "kube-api-access-p4x62") pod "f0097797-0e46-4510-83d6-5755d8a1242b" (UID: "f0097797-0e46-4510-83d6-5755d8a1242b"). InnerVolumeSpecName "kube-api-access-p4x62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:17:07.796637 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:07.796626    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f0097797-0e46-4510-83d6-5755d8a1242b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f0097797-0e46-4510-83d6-5755d8a1242b" (UID: "f0097797-0e46-4510-83d6-5755d8a1242b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 14:17:07.896223 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:07.896202    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0097797-0e46-4510-83d6-5755d8a1242b-istio-podinfo\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:17:07.896316 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:07.896224    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4x62\" (UniqueName: \"kubernetes.io/projected/f0097797-0e46-4510-83d6-5755d8a1242b-kube-api-access-p4x62\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:17:07.896316 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:07.896243    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0097797-0e46-4510-83d6-5755d8a1242b-istio-token\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:17:07.896316 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:07.896252    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0097797-0e46-4510-83d6-5755d8a1242b-istio-envoy\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:17:08.037157 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:08.037115    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:08.640725 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:08.640695    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x"
Sep 17 14:17:08.641178 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:08.640734    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x" event={"ID":"f0097797-0e46-4510-83d6-5755d8a1242b","Type":"ContainerDied","Data":"67fa9e26f9bc82ce753d4427020892ab6cfe595ad5db8a11dff0ae85f046c269"}
Sep 17 14:17:08.641178 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:08.640788    2560 scope.go:117] "RemoveContainer" containerID="e6fd7f3dd83c5a13915e00a0a1598597de428bba365c8b472cbdacfe49a2d789"
Sep 17 14:17:08.683371 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:08.683347    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x"]
Sep 17 14:17:08.687653 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:08.687630    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-4a43447e-openshift-default-6dfdc5c984-7hm8x"]
Sep 17 14:17:09.464589 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:09.464556    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:09.573393 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:09.573361    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0097797-0e46-4510-83d6-5755d8a1242b" path="/var/lib/kubelet/pods/f0097797-0e46-4510-83d6-5755d8a1242b/volumes"
Sep 17 14:17:10.133380 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:10.133341    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-767170/explicit-model-767170-kserve-9d9f5bc65-kxrd2" podUID="085bbbdf-8626-4a8c-be0b-f0ca98d7714d" containerName="main" containerID="cri-o://118222f4f1102823f4ddeab20aaeef717218e81feeb5538b1c1d365cee60d6d2" gracePeriod=45
Sep 17 14:17:10.276390 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:10.276359    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:10.345938 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:10.345906    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:10.402359 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:10.402332    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-767170/explicit-model-767170-kserve-9d9f5bc65-kxrd2"
Sep 17 14:17:10.459996 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:10.459957    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:10.520208 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:10.520183    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qmc85\" (UniqueName: \"kubernetes.io/projected/085bbbdf-8626-4a8c-be0b-f0ca98d7714d-kube-api-access-qmc85\") pod \"085bbbdf-8626-4a8c-be0b-f0ca98d7714d\" (UID: \"085bbbdf-8626-4a8c-be0b-f0ca98d7714d\") "
Sep 17 14:17:10.520339 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:10.520248    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/085bbbdf-8626-4a8c-be0b-f0ca98d7714d-model-cache\") pod \"085bbbdf-8626-4a8c-be0b-f0ca98d7714d\" (UID: \"085bbbdf-8626-4a8c-be0b-f0ca98d7714d\") "
Sep 17 14:17:10.520339 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:10.520326    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/085bbbdf-8626-4a8c-be0b-f0ca98d7714d-home\") pod \"085bbbdf-8626-4a8c-be0b-f0ca98d7714d\" (UID: \"085bbbdf-8626-4a8c-be0b-f0ca98d7714d\") "
Sep 17 14:17:10.520486 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:10.520355    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/085bbbdf-8626-4a8c-be0b-f0ca98d7714d-tmp-dir\") pod \"085bbbdf-8626-4a8c-be0b-f0ca98d7714d\" (UID: \"085bbbdf-8626-4a8c-be0b-f0ca98d7714d\") "
Sep 17 14:17:10.520486 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:10.520375    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/085bbbdf-8626-4a8c-be0b-f0ca98d7714d-dshm\") pod \"085bbbdf-8626-4a8c-be0b-f0ca98d7714d\" (UID: \"085bbbdf-8626-4a8c-be0b-f0ca98d7714d\") "
Sep 17 14:17:10.520486 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:10.520416    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/085bbbdf-8626-4a8c-be0b-f0ca98d7714d-tls-certs\") pod \"085bbbdf-8626-4a8c-be0b-f0ca98d7714d\" (UID: \"085bbbdf-8626-4a8c-be0b-f0ca98d7714d\") "
Sep 17 14:17:10.520655 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:10.520484    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/085bbbdf-8626-4a8c-be0b-f0ca98d7714d-model-cache" (OuterVolumeSpecName: "model-cache") pod "085bbbdf-8626-4a8c-be0b-f0ca98d7714d" (UID: "085bbbdf-8626-4a8c-be0b-f0ca98d7714d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:17:10.520655 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:10.520634    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/085bbbdf-8626-4a8c-be0b-f0ca98d7714d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "085bbbdf-8626-4a8c-be0b-f0ca98d7714d" (UID: "085bbbdf-8626-4a8c-be0b-f0ca98d7714d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:17:10.520655 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:10.520644    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/085bbbdf-8626-4a8c-be0b-f0ca98d7714d-home" (OuterVolumeSpecName: "home") pod "085bbbdf-8626-4a8c-be0b-f0ca98d7714d" (UID: "085bbbdf-8626-4a8c-be0b-f0ca98d7714d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:17:10.520838 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:10.520821    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/085bbbdf-8626-4a8c-be0b-f0ca98d7714d-model-cache\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:17:10.520899 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:10.520838    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/085bbbdf-8626-4a8c-be0b-f0ca98d7714d-home\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:17:10.520899 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:10.520849    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/085bbbdf-8626-4a8c-be0b-f0ca98d7714d-tmp-dir\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:17:10.522494 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:10.522454    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/085bbbdf-8626-4a8c-be0b-f0ca98d7714d-dshm" (OuterVolumeSpecName: "dshm") pod "085bbbdf-8626-4a8c-be0b-f0ca98d7714d" (UID: "085bbbdf-8626-4a8c-be0b-f0ca98d7714d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:17:10.522584 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:10.522539    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/085bbbdf-8626-4a8c-be0b-f0ca98d7714d-kube-api-access-qmc85" (OuterVolumeSpecName: "kube-api-access-qmc85") pod "085bbbdf-8626-4a8c-be0b-f0ca98d7714d" (UID: "085bbbdf-8626-4a8c-be0b-f0ca98d7714d"). InnerVolumeSpecName "kube-api-access-qmc85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:17:10.522584 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:10.522542    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/085bbbdf-8626-4a8c-be0b-f0ca98d7714d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "085bbbdf-8626-4a8c-be0b-f0ca98d7714d" (UID: "085bbbdf-8626-4a8c-be0b-f0ca98d7714d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:17:10.564188 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:10.564157    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:10.621962 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:10.621940    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/085bbbdf-8626-4a8c-be0b-f0ca98d7714d-dshm\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:17:10.621962 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:10.621960    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/085bbbdf-8626-4a8c-be0b-f0ca98d7714d-tls-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:17:10.622086 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:10.621974    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmc85\" (UniqueName: \"kubernetes.io/projected/085bbbdf-8626-4a8c-be0b-f0ca98d7714d-kube-api-access-qmc85\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:17:10.655862 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:10.655794    2560 generic.go:356] "Generic (PLEG): container finished" podID="085bbbdf-8626-4a8c-be0b-f0ca98d7714d" containerID="118222f4f1102823f4ddeab20aaeef717218e81feeb5538b1c1d365cee60d6d2" exitCode=0
Sep 17 14:17:10.655862 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:10.655833    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-767170/explicit-model-767170-kserve-9d9f5bc65-kxrd2" event={"ID":"085bbbdf-8626-4a8c-be0b-f0ca98d7714d","Type":"ContainerDied","Data":"118222f4f1102823f4ddeab20aaeef717218e81feeb5538b1c1d365cee60d6d2"}
Sep 17 14:17:10.655862 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:10.655854    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-767170/explicit-model-767170-kserve-9d9f5bc65-kxrd2"
Sep 17 14:17:10.656057 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:10.655888    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-767170/explicit-model-767170-kserve-9d9f5bc65-kxrd2" event={"ID":"085bbbdf-8626-4a8c-be0b-f0ca98d7714d","Type":"ContainerDied","Data":"d1884e0cd9bed2b665c388d2a91098f8c1b11b08bd6a2b1bbbb7e12037d04579"}
Sep 17 14:17:10.656057 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:10.655907    2560 scope.go:117] "RemoveContainer" containerID="118222f4f1102823f4ddeab20aaeef717218e81feeb5538b1c1d365cee60d6d2"
Sep 17 14:17:10.665947 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:10.665929    2560 scope.go:117] "RemoveContainer" containerID="118222f4f1102823f4ddeab20aaeef717218e81feeb5538b1c1d365cee60d6d2"
Sep 17 14:17:10.666263 ip-10-0-129-158 kubenswrapper[2560]: E0917 14:17:10.666240    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"118222f4f1102823f4ddeab20aaeef717218e81feeb5538b1c1d365cee60d6d2\": container with ID starting with 118222f4f1102823f4ddeab20aaeef717218e81feeb5538b1c1d365cee60d6d2 not found: ID does not exist" containerID="118222f4f1102823f4ddeab20aaeef717218e81feeb5538b1c1d365cee60d6d2"
Sep 17 14:17:10.666361 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:10.666269    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"118222f4f1102823f4ddeab20aaeef717218e81feeb5538b1c1d365cee60d6d2"} err="failed to get container status \"118222f4f1102823f4ddeab20aaeef717218e81feeb5538b1c1d365cee60d6d2\": rpc error: code = NotFound desc = could not find container \"118222f4f1102823f4ddeab20aaeef717218e81feeb5538b1c1d365cee60d6d2\": container with ID starting with 118222f4f1102823f4ddeab20aaeef717218e81feeb5538b1c1d365cee60d6d2 not found: ID does not exist"
Sep 17 14:17:10.669324 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:10.669298    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:10.690618 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:10.690597    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-767170/explicit-model-767170-kserve-9d9f5bc65-kxrd2"]
Sep 17 14:17:10.693451 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:10.693430    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-767170/explicit-model-767170-kserve-9d9f5bc65-kxrd2"]
Sep 17 14:17:11.570411 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:11.570381    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="085bbbdf-8626-4a8c-be0b-f0ca98d7714d" path="/var/lib/kubelet/pods/085bbbdf-8626-4a8c-be0b-f0ca98d7714d/volumes"
Sep 17 14:17:12.992499 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:12.992469    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:17.454960 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:17.454926    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:17.724666 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:17.724570    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:17.752286 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:17.752258    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s"]
Sep 17 14:17:17.752482 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:17.752438    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s" podUID="7ef3eba7-9ba6-46e2-aa43-cfa423635b70" containerName="istio-proxy" containerID="cri-o://2c3dd644b4cf8ff8db3452bb2f51d985342bb383464e0176d1c325c13f8ce683" gracePeriod=30
Sep 17 14:17:17.846375 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:17.846337    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:17.946312 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:17.946277    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:18.179818 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:18.179791    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:18.537846 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:18.537768    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:18.827293 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:18.827255    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:18.998704 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:18.998680    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:19.115921 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:19.115832    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l"]
Sep 17 14:17:19.116374 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:19.116353    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0097797-0e46-4510-83d6-5755d8a1242b" containerName="istio-proxy"
Sep 17 14:17:19.116374 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:19.116375    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0097797-0e46-4510-83d6-5755d8a1242b" containerName="istio-proxy"
Sep 17 14:17:19.116558 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:19.116391    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0097797-0e46-4510-83d6-5755d8a1242b" containerName="istio-proxy"
Sep 17 14:17:19.116558 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:19.116399    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0097797-0e46-4510-83d6-5755d8a1242b" containerName="istio-proxy"
Sep 17 14:17:19.116558 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:19.116421    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cbbbe5a-81dc-4bfa-9dd0-50eb76474f84" containerName="revoke-keys"
Sep 17 14:17:19.116558 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:19.116430    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cbbbe5a-81dc-4bfa-9dd0-50eb76474f84" containerName="revoke-keys"
Sep 17 14:17:19.116558 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:19.116450    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="085bbbdf-8626-4a8c-be0b-f0ca98d7714d" containerName="main"
Sep 17 14:17:19.116558 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:19.116474    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="085bbbdf-8626-4a8c-be0b-f0ca98d7714d" containerName="main"
Sep 17 14:17:19.116845 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:19.116597    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cbbbe5a-81dc-4bfa-9dd0-50eb76474f84" containerName="revoke-keys"
Sep 17 14:17:19.116845 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:19.116613    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0097797-0e46-4510-83d6-5755d8a1242b" containerName="istio-proxy"
Sep 17 14:17:19.116845 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:19.116625    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0097797-0e46-4510-83d6-5755d8a1242b" containerName="istio-proxy"
Sep 17 14:17:19.116845 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:19.116635    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="085bbbdf-8626-4a8c-be0b-f0ca98d7714d" containerName="main"
Sep 17 14:17:19.121432 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:19.121405    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l"
Sep 17 14:17:19.123639 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:19.123620    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-2b8275-openshift-default-dockercfg-8459v\""
Sep 17 14:17:19.133747 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:19.133728    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l"]
Sep 17 14:17:19.141956 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:19.141929    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:19.199629 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:19.199606    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52c3f621-606c-42b4-99b2-15dccb0ba8cc-istio-podinfo\") pod \"e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l\" (UID: \"52c3f621-606c-42b4-99b2-15dccb0ba8cc\") " pod="openshift-ingress/e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l"
Sep 17 14:17:19.199742 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:19.199649    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52c3f621-606c-42b4-99b2-15dccb0ba8cc-istio-data\") pod \"e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l\" (UID: \"52c3f621-606c-42b4-99b2-15dccb0ba8cc\") " pod="openshift-ingress/e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l"
Sep 17 14:17:19.199742 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:19.199709    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c3f621-606c-42b4-99b2-15dccb0ba8cc-credential-socket\") pod \"e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l\" (UID: \"52c3f621-606c-42b4-99b2-15dccb0ba8cc\") " pod="openshift-ingress/e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l"
Sep 17 14:17:19.199885 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:19.199748    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swgzh\" (UniqueName: \"kubernetes.io/projected/52c3f621-606c-42b4-99b2-15dccb0ba8cc-kube-api-access-swgzh\") pod \"e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l\" (UID: \"52c3f621-606c-42b4-99b2-15dccb0ba8cc\") " pod="openshift-ingress/e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l"
Sep 17 14:17:19.199885 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:19.199770    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52c3f621-606c-42b4-99b2-15dccb0ba8cc-istio-envoy\") pod \"e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l\" (UID: \"52c3f621-606c-42b4-99b2-15dccb0ba8cc\") " pod="openshift-ingress/e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l"
Sep 17 14:17:19.199885 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:19.199796    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52c3f621-606c-42b4-99b2-15dccb0ba8cc-istio-token\") pod \"e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l\" (UID: \"52c3f621-606c-42b4-99b2-15dccb0ba8cc\") " pod="openshift-ingress/e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l"
Sep 17 14:17:19.199885 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:19.199837    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c3f621-606c-42b4-99b2-15dccb0ba8cc-workload-socket\") pod \"e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l\" (UID: \"52c3f621-606c-42b4-99b2-15dccb0ba8cc\") " pod="openshift-ingress/e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l"
Sep 17 14:17:19.199885 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:19.199862    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52c3f621-606c-42b4-99b2-15dccb0ba8cc-workload-certs\") pod \"e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l\" (UID: \"52c3f621-606c-42b4-99b2-15dccb0ba8cc\") " pod="openshift-ingress/e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l"
Sep 17 14:17:19.200073 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:19.199936    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52c3f621-606c-42b4-99b2-15dccb0ba8cc-istiod-ca-cert\") pod \"e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l\" (UID: \"52c3f621-606c-42b4-99b2-15dccb0ba8cc\") " pod="openshift-ingress/e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l"
Sep 17 14:17:19.300602 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:19.300573    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52c3f621-606c-42b4-99b2-15dccb0ba8cc-istio-podinfo\") pod \"e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l\" (UID: \"52c3f621-606c-42b4-99b2-15dccb0ba8cc\") " pod="openshift-ingress/e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l"
Sep 17 14:17:19.300758 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:19.300624    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52c3f621-606c-42b4-99b2-15dccb0ba8cc-istio-data\") pod \"e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l\" (UID: \"52c3f621-606c-42b4-99b2-15dccb0ba8cc\") " pod="openshift-ingress/e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l"
Sep 17 14:17:19.300758 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:19.300664    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c3f621-606c-42b4-99b2-15dccb0ba8cc-credential-socket\") pod \"e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l\" (UID: \"52c3f621-606c-42b4-99b2-15dccb0ba8cc\") " pod="openshift-ingress/e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l"
Sep 17 14:17:19.300758 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:19.300689    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swgzh\" (UniqueName: \"kubernetes.io/projected/52c3f621-606c-42b4-99b2-15dccb0ba8cc-kube-api-access-swgzh\") pod \"e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l\" (UID: \"52c3f621-606c-42b4-99b2-15dccb0ba8cc\") " pod="openshift-ingress/e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l"
Sep 17 14:17:19.300758 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:19.300714    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52c3f621-606c-42b4-99b2-15dccb0ba8cc-istio-envoy\") pod \"e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l\" (UID: \"52c3f621-606c-42b4-99b2-15dccb0ba8cc\") " pod="openshift-ingress/e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l"
Sep 17 14:17:19.300758 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:19.300750    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52c3f621-606c-42b4-99b2-15dccb0ba8cc-istio-token\") pod \"e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l\" (UID: \"52c3f621-606c-42b4-99b2-15dccb0ba8cc\") " pod="openshift-ingress/e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l"
Sep 17 14:17:19.301035 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:19.300786    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c3f621-606c-42b4-99b2-15dccb0ba8cc-workload-socket\") pod \"e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l\" (UID: \"52c3f621-606c-42b4-99b2-15dccb0ba8cc\") " pod="openshift-ingress/e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l"
Sep 17 14:17:19.301035 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:19.300810    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52c3f621-606c-42b4-99b2-15dccb0ba8cc-workload-certs\") pod \"e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l\" (UID: \"52c3f621-606c-42b4-99b2-15dccb0ba8cc\") " pod="openshift-ingress/e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l"
Sep 17 14:17:19.301035 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:19.300910    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52c3f621-606c-42b4-99b2-15dccb0ba8cc-istiod-ca-cert\") pod \"e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l\" (UID: \"52c3f621-606c-42b4-99b2-15dccb0ba8cc\") " pod="openshift-ingress/e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l"
Sep 17 14:17:19.301185 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:19.301115    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c3f621-606c-42b4-99b2-15dccb0ba8cc-credential-socket\") pod \"e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l\" (UID: \"52c3f621-606c-42b4-99b2-15dccb0ba8cc\") " pod="openshift-ingress/e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l"
Sep 17 14:17:19.301262 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:19.301194    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52c3f621-606c-42b4-99b2-15dccb0ba8cc-istio-data\") pod \"e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l\" (UID: \"52c3f621-606c-42b4-99b2-15dccb0ba8cc\") " pod="openshift-ingress/e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l"
Sep 17 14:17:19.301262 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:19.301222    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52c3f621-606c-42b4-99b2-15dccb0ba8cc-workload-certs\") pod \"e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l\" (UID: \"52c3f621-606c-42b4-99b2-15dccb0ba8cc\") " pod="openshift-ingress/e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l"
Sep 17 14:17:19.301396 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:19.301375    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c3f621-606c-42b4-99b2-15dccb0ba8cc-workload-socket\") pod \"e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l\" (UID: \"52c3f621-606c-42b4-99b2-15dccb0ba8cc\") " pod="openshift-ingress/e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l"
Sep 17 14:17:19.301556 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:19.301538    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52c3f621-606c-42b4-99b2-15dccb0ba8cc-istiod-ca-cert\") pod \"e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l\" (UID: \"52c3f621-606c-42b4-99b2-15dccb0ba8cc\") " pod="openshift-ingress/e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l"
Sep 17 14:17:19.303074 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:19.303052    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52c3f621-606c-42b4-99b2-15dccb0ba8cc-istio-envoy\") pod \"e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l\" (UID: \"52c3f621-606c-42b4-99b2-15dccb0ba8cc\") " pod="openshift-ingress/e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l"
Sep 17 14:17:19.303179 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:19.303169    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52c3f621-606c-42b4-99b2-15dccb0ba8cc-istio-podinfo\") pod \"e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l\" (UID: \"52c3f621-606c-42b4-99b2-15dccb0ba8cc\") " pod="openshift-ingress/e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l"
Sep 17 14:17:19.316829 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:19.316800    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52c3f621-606c-42b4-99b2-15dccb0ba8cc-istio-token\") pod \"e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l\" (UID: \"52c3f621-606c-42b4-99b2-15dccb0ba8cc\") " pod="openshift-ingress/e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l"
Sep 17 14:17:19.316907 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:19.316827    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swgzh\" (UniqueName: \"kubernetes.io/projected/52c3f621-606c-42b4-99b2-15dccb0ba8cc-kube-api-access-swgzh\") pod \"e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l\" (UID: \"52c3f621-606c-42b4-99b2-15dccb0ba8cc\") " pod="openshift-ingress/e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l"
Sep 17 14:17:19.433395 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:19.433329    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l"
Sep 17 14:17:19.456431 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:19.456393    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:19.720797 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:19.720769    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l"]
Sep 17 14:17:19.722141 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:17:19.722114    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52c3f621_606c_42b4_99b2_15dccb0ba8cc.slice/crio-19ad9856825d027e52c654cb120a778ba5093bd540a09697633dbc6af5b8cd32 WatchSource:0}: Error finding container 19ad9856825d027e52c654cb120a778ba5093bd540a09697633dbc6af5b8cd32: Status 404 returned error can't find the container with id 19ad9856825d027e52c654cb120a778ba5093bd540a09697633dbc6af5b8cd32
Sep 17 14:17:19.724247 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:19.724209    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:17:19.724303 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:19.724294    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:17:19.724337 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:19.724322    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:17:20.374699 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:20.374664    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:20.481965 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:20.481925    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:20.577685 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:20.577658    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:20.725718 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:20.725621    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l" event={"ID":"52c3f621-606c-42b4-99b2-15dccb0ba8cc","Type":"ContainerStarted","Data":"e1c5e30ee457607e5c5f92e734776ddb9351f8d9c96432909da8d4577e46d19f"}
Sep 17 14:17:20.725718 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:20.725678    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l" event={"ID":"52c3f621-606c-42b4-99b2-15dccb0ba8cc","Type":"ContainerStarted","Data":"19ad9856825d027e52c654cb120a778ba5093bd540a09697633dbc6af5b8cd32"}
Sep 17 14:17:20.754775 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:20.754731    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l" podStartSLOduration=1.7547159049999999 podStartE2EDuration="1.754715905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:17:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:17:20.75279094 +0000 UTC m=+1309.694934585" watchObservedRunningTime="2026-09-17 14:17:20.754715905 +0000 UTC m=+1309.696859547"
Sep 17 14:17:21.433662 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:21.433626    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l"
Sep 17 14:17:21.439440 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:21.439415    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l"
Sep 17 14:17:21.732990 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:21.732912    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l"
Sep 17 14:17:21.733749 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:21.733733    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l"
Sep 17 14:17:22.003374 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:22.003296    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:22.486710 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:22.486668    2560 patch_prober.go:28] interesting pod/e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 17 14:17:22.486901 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:22.486735    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s" podUID="7ef3eba7-9ba6-46e2-aa43-cfa423635b70" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 14:17:23.003643 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:23.003619    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s"
Sep 17 14:17:23.139187 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:23.139163    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ef3eba7-9ba6-46e2-aa43-cfa423635b70-istiod-ca-cert\") pod \"7ef3eba7-9ba6-46e2-aa43-cfa423635b70\" (UID: \"7ef3eba7-9ba6-46e2-aa43-cfa423635b70\") "
Sep 17 14:17:23.139349 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:23.139214    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w2cs6\" (UniqueName: \"kubernetes.io/projected/7ef3eba7-9ba6-46e2-aa43-cfa423635b70-kube-api-access-w2cs6\") pod \"7ef3eba7-9ba6-46e2-aa43-cfa423635b70\" (UID: \"7ef3eba7-9ba6-46e2-aa43-cfa423635b70\") "
Sep 17 14:17:23.139349 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:23.139248    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ef3eba7-9ba6-46e2-aa43-cfa423635b70-istio-envoy\") pod \"7ef3eba7-9ba6-46e2-aa43-cfa423635b70\" (UID: \"7ef3eba7-9ba6-46e2-aa43-cfa423635b70\") "
Sep 17 14:17:23.139349 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:23.139271    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ef3eba7-9ba6-46e2-aa43-cfa423635b70-istio-podinfo\") pod \"7ef3eba7-9ba6-46e2-aa43-cfa423635b70\" (UID: \"7ef3eba7-9ba6-46e2-aa43-cfa423635b70\") "
Sep 17 14:17:23.139349 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:23.139340    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ef3eba7-9ba6-46e2-aa43-cfa423635b70-workload-socket\") pod \"7ef3eba7-9ba6-46e2-aa43-cfa423635b70\" (UID: \"7ef3eba7-9ba6-46e2-aa43-cfa423635b70\") "
Sep 17 14:17:23.139614 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:23.139369    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ef3eba7-9ba6-46e2-aa43-cfa423635b70-workload-certs\") pod \"7ef3eba7-9ba6-46e2-aa43-cfa423635b70\" (UID: \"7ef3eba7-9ba6-46e2-aa43-cfa423635b70\") "
Sep 17 14:17:23.139614 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:23.139402    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ef3eba7-9ba6-46e2-aa43-cfa423635b70-credential-socket\") pod \"7ef3eba7-9ba6-46e2-aa43-cfa423635b70\" (UID: \"7ef3eba7-9ba6-46e2-aa43-cfa423635b70\") "
Sep 17 14:17:23.139614 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:23.139450    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ef3eba7-9ba6-46e2-aa43-cfa423635b70-istio-token\") pod \"7ef3eba7-9ba6-46e2-aa43-cfa423635b70\" (UID: \"7ef3eba7-9ba6-46e2-aa43-cfa423635b70\") "
Sep 17 14:17:23.139614 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:23.139556    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ef3eba7-9ba6-46e2-aa43-cfa423635b70-istio-data\") pod \"7ef3eba7-9ba6-46e2-aa43-cfa423635b70\" (UID: \"7ef3eba7-9ba6-46e2-aa43-cfa423635b70\") "
Sep 17 14:17:23.139614 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:23.139576    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ef3eba7-9ba6-46e2-aa43-cfa423635b70-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7ef3eba7-9ba6-46e2-aa43-cfa423635b70" (UID: "7ef3eba7-9ba6-46e2-aa43-cfa423635b70"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:17:23.139873 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:23.139664    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ef3eba7-9ba6-46e2-aa43-cfa423635b70-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7ef3eba7-9ba6-46e2-aa43-cfa423635b70" (UID: "7ef3eba7-9ba6-46e2-aa43-cfa423635b70"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:17:23.139873 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:23.139682    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ef3eba7-9ba6-46e2-aa43-cfa423635b70-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7ef3eba7-9ba6-46e2-aa43-cfa423635b70" (UID: "7ef3eba7-9ba6-46e2-aa43-cfa423635b70"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:17:23.139873 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:23.139836    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ef3eba7-9ba6-46e2-aa43-cfa423635b70-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7ef3eba7-9ba6-46e2-aa43-cfa423635b70" (UID: "7ef3eba7-9ba6-46e2-aa43-cfa423635b70"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:17:23.140027 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:23.139989    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ef3eba7-9ba6-46e2-aa43-cfa423635b70-istio-data" (OuterVolumeSpecName: "istio-data") pod "7ef3eba7-9ba6-46e2-aa43-cfa423635b70" (UID: "7ef3eba7-9ba6-46e2-aa43-cfa423635b70"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:17:23.140027 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:23.140009    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ef3eba7-9ba6-46e2-aa43-cfa423635b70-istiod-ca-cert\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:17:23.140151 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:23.140029    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ef3eba7-9ba6-46e2-aa43-cfa423635b70-workload-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:17:23.140151 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:23.140039    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ef3eba7-9ba6-46e2-aa43-cfa423635b70-workload-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:17:23.140151 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:23.140048    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ef3eba7-9ba6-46e2-aa43-cfa423635b70-credential-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:17:23.141631 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:23.141603    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ef3eba7-9ba6-46e2-aa43-cfa423635b70-kube-api-access-w2cs6" (OuterVolumeSpecName: "kube-api-access-w2cs6") pod "7ef3eba7-9ba6-46e2-aa43-cfa423635b70" (UID: "7ef3eba7-9ba6-46e2-aa43-cfa423635b70"). InnerVolumeSpecName "kube-api-access-w2cs6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:17:23.141754 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:23.141635    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ef3eba7-9ba6-46e2-aa43-cfa423635b70-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7ef3eba7-9ba6-46e2-aa43-cfa423635b70" (UID: "7ef3eba7-9ba6-46e2-aa43-cfa423635b70"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 14:17:23.141937 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:23.141921    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ef3eba7-9ba6-46e2-aa43-cfa423635b70-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7ef3eba7-9ba6-46e2-aa43-cfa423635b70" (UID: "7ef3eba7-9ba6-46e2-aa43-cfa423635b70"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:17:23.141988 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:23.141932    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ef3eba7-9ba6-46e2-aa43-cfa423635b70-istio-token" (OuterVolumeSpecName: "istio-token") pod "7ef3eba7-9ba6-46e2-aa43-cfa423635b70" (UID: "7ef3eba7-9ba6-46e2-aa43-cfa423635b70"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:17:23.241270 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:23.241249    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ef3eba7-9ba6-46e2-aa43-cfa423635b70-istio-data\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:17:23.241270 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:23.241271    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2cs6\" (UniqueName: \"kubernetes.io/projected/7ef3eba7-9ba6-46e2-aa43-cfa423635b70-kube-api-access-w2cs6\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:17:23.241410 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:23.241280    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ef3eba7-9ba6-46e2-aa43-cfa423635b70-istio-envoy\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:17:23.241410 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:23.241289    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ef3eba7-9ba6-46e2-aa43-cfa423635b70-istio-podinfo\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:17:23.241410 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:23.241297    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ef3eba7-9ba6-46e2-aa43-cfa423635b70-istio-token\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:17:23.747850 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:23.747820    2560 generic.go:356] "Generic (PLEG): container finished" podID="7ef3eba7-9ba6-46e2-aa43-cfa423635b70" containerID="2c3dd644b4cf8ff8db3452bb2f51d985342bb383464e0176d1c325c13f8ce683" exitCode=0
Sep 17 14:17:23.748020 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:23.747871    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s"
Sep 17 14:17:23.748020 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:23.747894    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s" event={"ID":"7ef3eba7-9ba6-46e2-aa43-cfa423635b70","Type":"ContainerDied","Data":"2c3dd644b4cf8ff8db3452bb2f51d985342bb383464e0176d1c325c13f8ce683"}
Sep 17 14:17:23.748020 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:23.747939    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s" event={"ID":"7ef3eba7-9ba6-46e2-aa43-cfa423635b70","Type":"ContainerDied","Data":"62587722967d3ef3e6ef2f4de09742a859bcff5531dcfcee25bcf88793a02a60"}
Sep 17 14:17:23.748020 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:23.747962    2560 scope.go:117] "RemoveContainer" containerID="2c3dd644b4cf8ff8db3452bb2f51d985342bb383464e0176d1c325c13f8ce683"
Sep 17 14:17:23.757924 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:23.757907    2560 scope.go:117] "RemoveContainer" containerID="c978ec5b559db39a4409a204f9cf6cf3e92649464791e8bcbc01c3575e11fc7f"
Sep 17 14:17:23.766604 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:23.766580    2560 scope.go:117] "RemoveContainer" containerID="2c3dd644b4cf8ff8db3452bb2f51d985342bb383464e0176d1c325c13f8ce683"
Sep 17 14:17:23.766850 ip-10-0-129-158 kubenswrapper[2560]: E0917 14:17:23.766830    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c3dd644b4cf8ff8db3452bb2f51d985342bb383464e0176d1c325c13f8ce683\": container with ID starting with 2c3dd644b4cf8ff8db3452bb2f51d985342bb383464e0176d1c325c13f8ce683 not found: ID does not exist" containerID="2c3dd644b4cf8ff8db3452bb2f51d985342bb383464e0176d1c325c13f8ce683"
Sep 17 14:17:23.766924 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:23.766862    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c3dd644b4cf8ff8db3452bb2f51d985342bb383464e0176d1c325c13f8ce683"} err="failed to get container status \"2c3dd644b4cf8ff8db3452bb2f51d985342bb383464e0176d1c325c13f8ce683\": rpc error: code = NotFound desc = could not find container \"2c3dd644b4cf8ff8db3452bb2f51d985342bb383464e0176d1c325c13f8ce683\": container with ID starting with 2c3dd644b4cf8ff8db3452bb2f51d985342bb383464e0176d1c325c13f8ce683 not found: ID does not exist"
Sep 17 14:17:23.766924 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:23.766884    2560 scope.go:117] "RemoveContainer" containerID="c978ec5b559db39a4409a204f9cf6cf3e92649464791e8bcbc01c3575e11fc7f"
Sep 17 14:17:23.767168 ip-10-0-129-158 kubenswrapper[2560]: E0917 14:17:23.767150    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c978ec5b559db39a4409a204f9cf6cf3e92649464791e8bcbc01c3575e11fc7f\": container with ID starting with c978ec5b559db39a4409a204f9cf6cf3e92649464791e8bcbc01c3575e11fc7f not found: ID does not exist" containerID="c978ec5b559db39a4409a204f9cf6cf3e92649464791e8bcbc01c3575e11fc7f"
Sep 17 14:17:23.767211 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:23.767175    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c978ec5b559db39a4409a204f9cf6cf3e92649464791e8bcbc01c3575e11fc7f"} err="failed to get container status \"c978ec5b559db39a4409a204f9cf6cf3e92649464791e8bcbc01c3575e11fc7f\": rpc error: code = NotFound desc = could not find container \"c978ec5b559db39a4409a204f9cf6cf3e92649464791e8bcbc01c3575e11fc7f\": container with ID starting with c978ec5b559db39a4409a204f9cf6cf3e92649464791e8bcbc01c3575e11fc7f not found: ID does not exist"
Sep 17 14:17:23.771557 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:23.771535    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s"]
Sep 17 14:17:23.784777 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:23.784757    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-767170-openshift-default-65b9b6c956-mrt9s"]
Sep 17 14:17:23.934107 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:23.934025    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:24.221736 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:24.221645    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:25.232059 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:25.232024    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:25.570006 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:25.569978    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ef3eba7-9ba6-46e2-aa43-cfa423635b70" path="/var/lib/kubelet/pods/7ef3eba7-9ba6-46e2-aa43-cfa423635b70/volumes"
Sep 17 14:17:26.081118 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:26.081085    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:26.966410 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:26.966373    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:27.064347 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:27.064307    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:27.163820 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:27.163777    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:27.490403 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:27.490369    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:27.590280 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:27.590236    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:41.147038 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:41.147003    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-079b38/e2e-conflict-a-9812c892-kserve-5655867767-qkchj"]
Sep 17 14:17:41.147509 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:41.147495    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ef3eba7-9ba6-46e2-aa43-cfa423635b70" containerName="istio-proxy"
Sep 17 14:17:41.147509 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:41.147510    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ef3eba7-9ba6-46e2-aa43-cfa423635b70" containerName="istio-proxy"
Sep 17 14:17:41.147611 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:41.147602    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ef3eba7-9ba6-46e2-aa43-cfa423635b70" containerName="istio-proxy"
Sep 17 14:17:41.147651 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:41.147612    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ef3eba7-9ba6-46e2-aa43-cfa423635b70" containerName="istio-proxy"
Sep 17 14:17:41.151994 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:41.151977    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-079b38/e2e-conflict-a-9812c892-kserve-5655867767-qkchj"
Sep 17 14:17:41.154627 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:41.154607    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-079b38\"/\"kube-root-ca.crt\""
Sep 17 14:17:41.155182 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:41.155163    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-079b38\"/\"openshift-service-ca.crt\""
Sep 17 14:17:41.155182 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:41.155173    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-079b38\"/\"default-dockercfg-xlhg2\""
Sep 17 14:17:41.155347 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:41.155258    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-079b38\"/\"e2e-conflict-a-9812c892-kserve-self-signed-certs\""
Sep 17 14:17:41.164788 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:41.164764    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-079b38/e2e-conflict-a-9812c892-kserve-5655867767-qkchj"]
Sep 17 14:17:41.211811 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:41.211772    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e903702-6868-403e-aabb-94e3faac371e-model-cache\") pod \"e2e-conflict-a-9812c892-kserve-5655867767-qkchj\" (UID: \"6e903702-6868-403e-aabb-94e3faac371e\") " pod="e2e-models-e2e-worker-w4-079b38/e2e-conflict-a-9812c892-kserve-5655867767-qkchj"
Sep 17 14:17:41.212018 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:41.211850    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e903702-6868-403e-aabb-94e3faac371e-dshm\") pod \"e2e-conflict-a-9812c892-kserve-5655867767-qkchj\" (UID: \"6e903702-6868-403e-aabb-94e3faac371e\") " pod="e2e-models-e2e-worker-w4-079b38/e2e-conflict-a-9812c892-kserve-5655867767-qkchj"
Sep 17 14:17:41.212018 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:41.211886    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e903702-6868-403e-aabb-94e3faac371e-tls-certs\") pod \"e2e-conflict-a-9812c892-kserve-5655867767-qkchj\" (UID: \"6e903702-6868-403e-aabb-94e3faac371e\") " pod="e2e-models-e2e-worker-w4-079b38/e2e-conflict-a-9812c892-kserve-5655867767-qkchj"
Sep 17 14:17:41.212018 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:41.211922    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4m9c\" (UniqueName: \"kubernetes.io/projected/6e903702-6868-403e-aabb-94e3faac371e-kube-api-access-l4m9c\") pod \"e2e-conflict-a-9812c892-kserve-5655867767-qkchj\" (UID: \"6e903702-6868-403e-aabb-94e3faac371e\") " pod="e2e-models-e2e-worker-w4-079b38/e2e-conflict-a-9812c892-kserve-5655867767-qkchj"
Sep 17 14:17:41.212018 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:41.211979    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e903702-6868-403e-aabb-94e3faac371e-tmp-dir\") pod \"e2e-conflict-a-9812c892-kserve-5655867767-qkchj\" (UID: \"6e903702-6868-403e-aabb-94e3faac371e\") " pod="e2e-models-e2e-worker-w4-079b38/e2e-conflict-a-9812c892-kserve-5655867767-qkchj"
Sep 17 14:17:41.212265 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:41.212091    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e903702-6868-403e-aabb-94e3faac371e-home\") pod \"e2e-conflict-a-9812c892-kserve-5655867767-qkchj\" (UID: \"6e903702-6868-403e-aabb-94e3faac371e\") " pod="e2e-models-e2e-worker-w4-079b38/e2e-conflict-a-9812c892-kserve-5655867767-qkchj"
Sep 17 14:17:41.233539 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:41.233512    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:41.312690 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:41.312654    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e903702-6868-403e-aabb-94e3faac371e-tmp-dir\") pod \"e2e-conflict-a-9812c892-kserve-5655867767-qkchj\" (UID: \"6e903702-6868-403e-aabb-94e3faac371e\") " pod="e2e-models-e2e-worker-w4-079b38/e2e-conflict-a-9812c892-kserve-5655867767-qkchj"
Sep 17 14:17:41.312865 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:41.312738    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e903702-6868-403e-aabb-94e3faac371e-home\") pod \"e2e-conflict-a-9812c892-kserve-5655867767-qkchj\" (UID: \"6e903702-6868-403e-aabb-94e3faac371e\") " pod="e2e-models-e2e-worker-w4-079b38/e2e-conflict-a-9812c892-kserve-5655867767-qkchj"
Sep 17 14:17:41.312865 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:41.312768    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e903702-6868-403e-aabb-94e3faac371e-model-cache\") pod \"e2e-conflict-a-9812c892-kserve-5655867767-qkchj\" (UID: \"6e903702-6868-403e-aabb-94e3faac371e\") " pod="e2e-models-e2e-worker-w4-079b38/e2e-conflict-a-9812c892-kserve-5655867767-qkchj"
Sep 17 14:17:41.312865 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:41.312807    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e903702-6868-403e-aabb-94e3faac371e-dshm\") pod \"e2e-conflict-a-9812c892-kserve-5655867767-qkchj\" (UID: \"6e903702-6868-403e-aabb-94e3faac371e\") " pod="e2e-models-e2e-worker-w4-079b38/e2e-conflict-a-9812c892-kserve-5655867767-qkchj"
Sep 17 14:17:41.312865 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:41.312832    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e903702-6868-403e-aabb-94e3faac371e-tls-certs\") pod \"e2e-conflict-a-9812c892-kserve-5655867767-qkchj\" (UID: \"6e903702-6868-403e-aabb-94e3faac371e\") " pod="e2e-models-e2e-worker-w4-079b38/e2e-conflict-a-9812c892-kserve-5655867767-qkchj"
Sep 17 14:17:41.313044 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:41.312881    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4m9c\" (UniqueName: \"kubernetes.io/projected/6e903702-6868-403e-aabb-94e3faac371e-kube-api-access-l4m9c\") pod \"e2e-conflict-a-9812c892-kserve-5655867767-qkchj\" (UID: \"6e903702-6868-403e-aabb-94e3faac371e\") " pod="e2e-models-e2e-worker-w4-079b38/e2e-conflict-a-9812c892-kserve-5655867767-qkchj"
Sep 17 14:17:41.313140 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:41.313117    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e903702-6868-403e-aabb-94e3faac371e-tmp-dir\") pod \"e2e-conflict-a-9812c892-kserve-5655867767-qkchj\" (UID: \"6e903702-6868-403e-aabb-94e3faac371e\") " pod="e2e-models-e2e-worker-w4-079b38/e2e-conflict-a-9812c892-kserve-5655867767-qkchj"
Sep 17 14:17:41.313185 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:41.313121    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e903702-6868-403e-aabb-94e3faac371e-home\") pod \"e2e-conflict-a-9812c892-kserve-5655867767-qkchj\" (UID: \"6e903702-6868-403e-aabb-94e3faac371e\") " pod="e2e-models-e2e-worker-w4-079b38/e2e-conflict-a-9812c892-kserve-5655867767-qkchj"
Sep 17 14:17:41.313229 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:41.313177    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e903702-6868-403e-aabb-94e3faac371e-model-cache\") pod \"e2e-conflict-a-9812c892-kserve-5655867767-qkchj\" (UID: \"6e903702-6868-403e-aabb-94e3faac371e\") " pod="e2e-models-e2e-worker-w4-079b38/e2e-conflict-a-9812c892-kserve-5655867767-qkchj"
Sep 17 14:17:41.315213 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:41.315188    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e903702-6868-403e-aabb-94e3faac371e-dshm\") pod \"e2e-conflict-a-9812c892-kserve-5655867767-qkchj\" (UID: \"6e903702-6868-403e-aabb-94e3faac371e\") " pod="e2e-models-e2e-worker-w4-079b38/e2e-conflict-a-9812c892-kserve-5655867767-qkchj"
Sep 17 14:17:41.315436 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:41.315405    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e903702-6868-403e-aabb-94e3faac371e-tls-certs\") pod \"e2e-conflict-a-9812c892-kserve-5655867767-qkchj\" (UID: \"6e903702-6868-403e-aabb-94e3faac371e\") " pod="e2e-models-e2e-worker-w4-079b38/e2e-conflict-a-9812c892-kserve-5655867767-qkchj"
Sep 17 14:17:41.322106 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:41.322080    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4m9c\" (UniqueName: \"kubernetes.io/projected/6e903702-6868-403e-aabb-94e3faac371e-kube-api-access-l4m9c\") pod \"e2e-conflict-a-9812c892-kserve-5655867767-qkchj\" (UID: \"6e903702-6868-403e-aabb-94e3faac371e\") " pod="e2e-models-e2e-worker-w4-079b38/e2e-conflict-a-9812c892-kserve-5655867767-qkchj"
Sep 17 14:17:41.461777 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:41.461688    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-079b38/e2e-conflict-a-9812c892-kserve-5655867767-qkchj"
Sep 17 14:17:41.741316 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:41.741293    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-079b38/e2e-conflict-a-9812c892-kserve-5655867767-qkchj"]
Sep 17 14:17:41.742271 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:17:41.742242    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6e903702_6868_403e_aabb_94e3faac371e.slice/crio-278b0805148f69df0feeb6a060132ad2b223fb81956f284d283cc669935204a5 WatchSource:0}: Error finding container 278b0805148f69df0feeb6a060132ad2b223fb81956f284d283cc669935204a5: Status 404 returned error can't find the container with id 278b0805148f69df0feeb6a060132ad2b223fb81956f284d283cc669935204a5
Sep 17 14:17:41.877528 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:41.877492    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-079b38/e2e-conflict-a-9812c892-kserve-5655867767-qkchj" event={"ID":"6e903702-6868-403e-aabb-94e3faac371e","Type":"ContainerStarted","Data":"4c59fb512c905c00ee41a5e8012c7107e56d3819df79b184c1358d5dc31e85c1"}
Sep 17 14:17:41.877689 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:41.877539    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-079b38/e2e-conflict-a-9812c892-kserve-5655867767-qkchj" event={"ID":"6e903702-6868-403e-aabb-94e3faac371e","Type":"ContainerStarted","Data":"278b0805148f69df0feeb6a060132ad2b223fb81956f284d283cc669935204a5"}
Sep 17 14:17:41.897622 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:41.897574    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-079b38/e2e-conflict-a-9812c892-kserve-5655867767-qkchj" podStartSLOduration=0.897558935 podStartE2EDuration="897.558935ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:17:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:17:41.894843831 +0000 UTC m=+1330.836987481" watchObservedRunningTime="2026-09-17 14:17:41.897558935 +0000 UTC m=+1330.839702597"
Sep 17 14:17:50.679745 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:50.679707    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:51.461809 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:51.461775    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-079b38/e2e-conflict-a-9812c892-kserve-5655867767-qkchj"
Sep 17 14:17:51.462000 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:51.461852    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-079b38/e2e-conflict-a-9812c892-kserve-5655867767-qkchj"
Sep 17 14:17:51.474273 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:51.474244    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-079b38/e2e-conflict-a-9812c892-kserve-5655867767-qkchj"
Sep 17 14:17:51.642683 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:51.642650    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:51.733888 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:51.733865    2560 patch_prober.go:28] interesting pod/e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Sep 17 14:17:51.734165 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:51.733923    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l" podUID="52c3f621-606c-42b4-99b2-15dccb0ba8cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Sep 17 14:17:51.947823 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:51.947751    2560 generic.go:356] "Generic (PLEG): container finished" podID="52c3f621-606c-42b4-99b2-15dccb0ba8cc" containerID="e1c5e30ee457607e5c5f92e734776ddb9351f8d9c96432909da8d4577e46d19f" exitCode=0
Sep 17 14:17:51.947965 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:51.947821    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l" event={"ID":"52c3f621-606c-42b4-99b2-15dccb0ba8cc","Type":"ContainerDied","Data":"e1c5e30ee457607e5c5f92e734776ddb9351f8d9c96432909da8d4577e46d19f"}
Sep 17 14:17:51.948378 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:51.948346    2560 scope.go:117] "RemoveContainer" containerID="e1c5e30ee457607e5c5f92e734776ddb9351f8d9c96432909da8d4577e46d19f"
Sep 17 14:17:51.949538 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:51.949514    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:17:51.949607 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:51.949583    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:17:51.949643 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:51.949617    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:17:51.963574 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:51.963548    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-079b38/e2e-conflict-a-9812c892-kserve-5655867767-qkchj"
Sep 17 14:17:52.633706 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:52.633671    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:52.801062 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:52.801027    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:52.981411 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:52.981307    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l" event={"ID":"52c3f621-606c-42b4-99b2-15dccb0ba8cc","Type":"ContainerStarted","Data":"c557480e3e9fab689bf05e9b2ccfc2303bc816c5ae3167516e7de1ce243f9f8b"}
Sep 17 14:17:53.136295 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:53.136246    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:53.433558 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:53.433521    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l"
Sep 17 14:17:53.433798 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:53.433771    2560 patch_prober.go:28] interesting pod/e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Sep 17 14:17:53.433858 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:53.433834    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l" podUID="52c3f621-606c-42b4-99b2-15dccb0ba8cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Sep 17 14:17:53.484138 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:53.484109    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt"]
Sep 17 14:17:53.484597 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:53.484583    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ef3eba7-9ba6-46e2-aa43-cfa423635b70" containerName="istio-proxy"
Sep 17 14:17:53.484661 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:53.484599    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ef3eba7-9ba6-46e2-aa43-cfa423635b70" containerName="istio-proxy"
Sep 17 14:17:53.488314 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:53.488297    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt"
Sep 17 14:17:53.492333 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:53.492307    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-6e2bb9-openshift-default-dockercfg-thlht\""
Sep 17 14:17:53.515696 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:53.515667    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt"]
Sep 17 14:17:53.529970 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:53.529940    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:53.643567 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:53.643535    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c66fc21d-4519-4293-bc1f-4e9ae8fe2564-istio-token\") pod \"e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt\" (UID: \"c66fc21d-4519-4293-bc1f-4e9ae8fe2564\") " pod="openshift-ingress/e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt"
Sep 17 14:17:53.643748 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:53.643599    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7mpg\" (UniqueName: \"kubernetes.io/projected/c66fc21d-4519-4293-bc1f-4e9ae8fe2564-kube-api-access-l7mpg\") pod \"e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt\" (UID: \"c66fc21d-4519-4293-bc1f-4e9ae8fe2564\") " pod="openshift-ingress/e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt"
Sep 17 14:17:53.643748 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:53.643675    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c66fc21d-4519-4293-bc1f-4e9ae8fe2564-istio-data\") pod \"e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt\" (UID: \"c66fc21d-4519-4293-bc1f-4e9ae8fe2564\") " pod="openshift-ingress/e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt"
Sep 17 14:17:53.643748 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:53.643694    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c66fc21d-4519-4293-bc1f-4e9ae8fe2564-credential-socket\") pod \"e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt\" (UID: \"c66fc21d-4519-4293-bc1f-4e9ae8fe2564\") " pod="openshift-ingress/e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt"
Sep 17 14:17:53.643748 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:53.643736    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c66fc21d-4519-4293-bc1f-4e9ae8fe2564-istio-podinfo\") pod \"e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt\" (UID: \"c66fc21d-4519-4293-bc1f-4e9ae8fe2564\") " pod="openshift-ingress/e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt"
Sep 17 14:17:53.643994 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:53.643757    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c66fc21d-4519-4293-bc1f-4e9ae8fe2564-istiod-ca-cert\") pod \"e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt\" (UID: \"c66fc21d-4519-4293-bc1f-4e9ae8fe2564\") " pod="openshift-ingress/e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt"
Sep 17 14:17:53.643994 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:53.643841    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c66fc21d-4519-4293-bc1f-4e9ae8fe2564-workload-certs\") pod \"e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt\" (UID: \"c66fc21d-4519-4293-bc1f-4e9ae8fe2564\") " pod="openshift-ingress/e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt"
Sep 17 14:17:53.643994 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:53.643883    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c66fc21d-4519-4293-bc1f-4e9ae8fe2564-istio-envoy\") pod \"e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt\" (UID: \"c66fc21d-4519-4293-bc1f-4e9ae8fe2564\") " pod="openshift-ingress/e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt"
Sep 17 14:17:53.643994 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:53.643915    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c66fc21d-4519-4293-bc1f-4e9ae8fe2564-workload-socket\") pod \"e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt\" (UID: \"c66fc21d-4519-4293-bc1f-4e9ae8fe2564\") " pod="openshift-ingress/e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt"
Sep 17 14:17:53.744632 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:53.744544    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c66fc21d-4519-4293-bc1f-4e9ae8fe2564-istio-podinfo\") pod \"e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt\" (UID: \"c66fc21d-4519-4293-bc1f-4e9ae8fe2564\") " pod="openshift-ingress/e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt"
Sep 17 14:17:53.744632 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:53.744577    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c66fc21d-4519-4293-bc1f-4e9ae8fe2564-istiod-ca-cert\") pod \"e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt\" (UID: \"c66fc21d-4519-4293-bc1f-4e9ae8fe2564\") " pod="openshift-ingress/e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt"
Sep 17 14:17:53.744632 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:53.744626    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c66fc21d-4519-4293-bc1f-4e9ae8fe2564-workload-certs\") pod \"e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt\" (UID: \"c66fc21d-4519-4293-bc1f-4e9ae8fe2564\") " pod="openshift-ingress/e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt"
Sep 17 14:17:53.744892 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:53.744652    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c66fc21d-4519-4293-bc1f-4e9ae8fe2564-istio-envoy\") pod \"e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt\" (UID: \"c66fc21d-4519-4293-bc1f-4e9ae8fe2564\") " pod="openshift-ingress/e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt"
Sep 17 14:17:53.744892 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:53.744672    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c66fc21d-4519-4293-bc1f-4e9ae8fe2564-workload-socket\") pod \"e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt\" (UID: \"c66fc21d-4519-4293-bc1f-4e9ae8fe2564\") " pod="openshift-ingress/e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt"
Sep 17 14:17:53.744892 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:53.744745    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c66fc21d-4519-4293-bc1f-4e9ae8fe2564-istio-token\") pod \"e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt\" (UID: \"c66fc21d-4519-4293-bc1f-4e9ae8fe2564\") " pod="openshift-ingress/e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt"
Sep 17 14:17:53.744892 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:53.744794    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7mpg\" (UniqueName: \"kubernetes.io/projected/c66fc21d-4519-4293-bc1f-4e9ae8fe2564-kube-api-access-l7mpg\") pod \"e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt\" (UID: \"c66fc21d-4519-4293-bc1f-4e9ae8fe2564\") " pod="openshift-ingress/e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt"
Sep 17 14:17:53.744892 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:53.744869    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c66fc21d-4519-4293-bc1f-4e9ae8fe2564-istio-data\") pod \"e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt\" (UID: \"c66fc21d-4519-4293-bc1f-4e9ae8fe2564\") " pod="openshift-ingress/e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt"
Sep 17 14:17:53.745138 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:53.744898    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c66fc21d-4519-4293-bc1f-4e9ae8fe2564-credential-socket\") pod \"e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt\" (UID: \"c66fc21d-4519-4293-bc1f-4e9ae8fe2564\") " pod="openshift-ingress/e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt"
Sep 17 14:17:53.745138 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:53.745090    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c66fc21d-4519-4293-bc1f-4e9ae8fe2564-workload-certs\") pod \"e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt\" (UID: \"c66fc21d-4519-4293-bc1f-4e9ae8fe2564\") " pod="openshift-ingress/e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt"
Sep 17 14:17:53.745255 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:53.745237    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c66fc21d-4519-4293-bc1f-4e9ae8fe2564-credential-socket\") pod \"e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt\" (UID: \"c66fc21d-4519-4293-bc1f-4e9ae8fe2564\") " pod="openshift-ingress/e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt"
Sep 17 14:17:53.745317 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:53.745273    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c66fc21d-4519-4293-bc1f-4e9ae8fe2564-istio-data\") pod \"e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt\" (UID: \"c66fc21d-4519-4293-bc1f-4e9ae8fe2564\") " pod="openshift-ingress/e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt"
Sep 17 14:17:53.745383 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:53.745324    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c66fc21d-4519-4293-bc1f-4e9ae8fe2564-workload-socket\") pod \"e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt\" (UID: \"c66fc21d-4519-4293-bc1f-4e9ae8fe2564\") " pod="openshift-ingress/e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt"
Sep 17 14:17:53.745675 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:53.745657    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c66fc21d-4519-4293-bc1f-4e9ae8fe2564-istiod-ca-cert\") pod \"e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt\" (UID: \"c66fc21d-4519-4293-bc1f-4e9ae8fe2564\") " pod="openshift-ingress/e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt"
Sep 17 14:17:53.747128 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:53.747108    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c66fc21d-4519-4293-bc1f-4e9ae8fe2564-istio-envoy\") pod \"e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt\" (UID: \"c66fc21d-4519-4293-bc1f-4e9ae8fe2564\") " pod="openshift-ingress/e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt"
Sep 17 14:17:53.747220 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:53.747200    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c66fc21d-4519-4293-bc1f-4e9ae8fe2564-istio-podinfo\") pod \"e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt\" (UID: \"c66fc21d-4519-4293-bc1f-4e9ae8fe2564\") " pod="openshift-ingress/e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt"
Sep 17 14:17:53.779989 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:53.779956    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c66fc21d-4519-4293-bc1f-4e9ae8fe2564-istio-token\") pod \"e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt\" (UID: \"c66fc21d-4519-4293-bc1f-4e9ae8fe2564\") " pod="openshift-ingress/e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt"
Sep 17 14:17:53.800031 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:53.800011    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7mpg\" (UniqueName: \"kubernetes.io/projected/c66fc21d-4519-4293-bc1f-4e9ae8fe2564-kube-api-access-l7mpg\") pod \"e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt\" (UID: \"c66fc21d-4519-4293-bc1f-4e9ae8fe2564\") " pod="openshift-ingress/e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt"
Sep 17 14:17:53.807665 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:53.807636    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:53.903353 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:53.903313    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:54.099213 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:54.099185    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt"
Sep 17 14:17:54.407599 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:17:54.407571    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc66fc21d_4519_4293_bc1f_4e9ae8fe2564.slice/crio-b790898beb46d08725d4e68fc070a59b4c8233882d9a5b450a4c42a99d8359e7 WatchSource:0}: Error finding container b790898beb46d08725d4e68fc070a59b4c8233882d9a5b450a4c42a99d8359e7: Status 404 returned error can't find the container with id b790898beb46d08725d4e68fc070a59b4c8233882d9a5b450a4c42a99d8359e7
Sep 17 14:17:54.409737 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:54.409706    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt"]
Sep 17 14:17:54.410253 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:54.410225    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:17:54.410301 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:54.410290    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:17:54.410336 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:54.410320    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:17:54.433903 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:54.433878    2560 patch_prober.go:28] interesting pod/e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Sep 17 14:17:54.433996 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:54.433951    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l" podUID="52c3f621-606c-42b4-99b2-15dccb0ba8cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Sep 17 14:17:54.690277 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:54.690146    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:54.996145 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:54.996048    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt" event={"ID":"c66fc21d-4519-4293-bc1f-4e9ae8fe2564","Type":"ContainerStarted","Data":"9880e4126cffa56d0bbb6f0975de61ee8898a364be70441398c8c624ce64a287"}
Sep 17 14:17:54.996145 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:54.996115    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt" event={"ID":"c66fc21d-4519-4293-bc1f-4e9ae8fe2564","Type":"ContainerStarted","Data":"b790898beb46d08725d4e68fc070a59b4c8233882d9a5b450a4c42a99d8359e7"}
Sep 17 14:17:55.024395 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:55.024348    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt" podStartSLOduration=2.024333339 podStartE2EDuration="2.024333339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:17:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:17:55.02055716 +0000 UTC m=+1343.962700809" watchObservedRunningTime="2026-09-17 14:17:55.024333339 +0000 UTC m=+1343.966476985"
Sep 17 14:17:55.099757 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:55.099731    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt"
Sep 17 14:17:55.105226 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:55.105201    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt"
Sep 17 14:17:55.434262 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:55.434227    2560 patch_prober.go:28] interesting pod/e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Sep 17 14:17:55.434437 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:55.434280    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l" podUID="52c3f621-606c-42b4-99b2-15dccb0ba8cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Sep 17 14:17:55.747668 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:55.747583    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:56.002336 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:56.002242    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt"
Sep 17 14:17:56.003238 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:56.003217    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt"
Sep 17 14:17:56.434517 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:56.434481    2560 patch_prober.go:28] interesting pod/e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Sep 17 14:17:56.434700 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:56.434544    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l" podUID="52c3f621-606c-42b4-99b2-15dccb0ba8cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Sep 17 14:17:56.611324 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:56.611287    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:57.200956 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:57.200918    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:57.287383 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:57.287334    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:57.433713 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:57.433682    2560 patch_prober.go:28] interesting pod/e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Sep 17 14:17:57.433866 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:57.433730    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l" podUID="52c3f621-606c-42b4-99b2-15dccb0ba8cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Sep 17 14:17:57.684234 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:57.684187    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:58.437897 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:58.437869    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l"
Sep 17 14:17:58.438407 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:58.438162    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l"
Sep 17 14:17:58.438986 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:58.438964    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l"
Sep 17 14:17:58.591983 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:58.591944    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:59.454890 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:17:59.454853    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:00.488469 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:00.488433    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:01.303242 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:01.303202    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:01.706254 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:01.706175    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:01.808902 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:01.808867    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:02.196505 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:02.196450    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:02.386822 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:02.386791    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:02.484115 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:02.484035    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:02.823227 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:02.823199    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:02.926545 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:02.926511    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:24.876284 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:24.876247    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:25.638362 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:25.638330    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:26.475138 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:26.475102    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:27.201514 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:27.201480    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:27.557172 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:27.557116    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:27.955392 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:27.955310    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7774-e0678b"]
Sep 17 14:18:27.961601 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:27.961575    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7774-e0678b"
Sep 17 14:18:27.967260 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:27.967231    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7774-e0678b"]
Sep 17 14:18:27.969702 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:27.969683    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-f8z2f\""
Sep 17 14:18:28.109770 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:28.109742    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf2hk\" (UniqueName: \"kubernetes.io/projected/ef7db703-e60c-4ba4-97cf-6865c11e7841-kube-api-access-lf2hk\") pod \"test-curl-7774-e0678b\" (UID: \"ef7db703-e60c-4ba4-97cf-6865c11e7841\") " pod="opendatahub/test-curl-7774-e0678b"
Sep 17 14:18:28.210792 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:28.210703    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf2hk\" (UniqueName: \"kubernetes.io/projected/ef7db703-e60c-4ba4-97cf-6865c11e7841-kube-api-access-lf2hk\") pod \"test-curl-7774-e0678b\" (UID: \"ef7db703-e60c-4ba4-97cf-6865c11e7841\") " pod="opendatahub/test-curl-7774-e0678b"
Sep 17 14:18:28.219608 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:28.219586    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf2hk\" (UniqueName: \"kubernetes.io/projected/ef7db703-e60c-4ba4-97cf-6865c11e7841-kube-api-access-lf2hk\") pod \"test-curl-7774-e0678b\" (UID: \"ef7db703-e60c-4ba4-97cf-6865c11e7841\") " pod="opendatahub/test-curl-7774-e0678b"
Sep 17 14:18:28.284740 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:28.284708    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7774-e0678b"
Sep 17 14:18:28.382449 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:28.382361    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:28.559008 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:28.558977    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7774-e0678b"]
Sep 17 14:18:28.559415 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:18:28.558999    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podef7db703_e60c_4ba4_97cf_6865c11e7841.slice/crio-3956519ccdcc0651f2bc5a6237d5cf25eb39e2088de5d8bb331a3e80b26b9261 WatchSource:0}: Error finding container 3956519ccdcc0651f2bc5a6237d5cf25eb39e2088de5d8bb331a3e80b26b9261: Status 404 returned error can't find the container with id 3956519ccdcc0651f2bc5a6237d5cf25eb39e2088de5d8bb331a3e80b26b9261
Sep 17 14:18:28.560988 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:28.560970    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 14:18:29.225193 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:29.225150    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7774-e0678b" event={"ID":"ef7db703-e60c-4ba4-97cf-6865c11e7841","Type":"ContainerStarted","Data":"3956519ccdcc0651f2bc5a6237d5cf25eb39e2088de5d8bb331a3e80b26b9261"}
Sep 17 14:18:29.456728 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:29.456675    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:30.147629 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:30.147587    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:30.233930 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:30.233892    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7774-e0678b" event={"ID":"ef7db703-e60c-4ba4-97cf-6865c11e7841","Type":"ContainerStarted","Data":"2bd063a43fa40c7e1bca62adec4749518f040698b7d7a8e19693bf904a9aa4d2"}
Sep 17 14:18:30.255828 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:30.255785    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7774-e0678b" podStartSLOduration=2.102788664 podStartE2EDuration="3.255767405s" totalImagesPullingTime="1.152978741s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:18:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:18:30.253181909 +0000 UTC m=+1379.195325554" watchObservedRunningTime="2026-09-17 14:18:30.255767405 +0000 UTC m=+1379.197911102"
Sep 17 14:18:30.490869 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:30.490838    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-96cfc6-5bn2d"]
Sep 17 14:18:30.494740 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:30.494722    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-96cfc6-5bn2d"
Sep 17 14:18:30.496985 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:30.496964    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rp9ww\""
Sep 17 14:18:30.497093 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:30.497049    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 14:18:30.507082 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:30.507056    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-96cfc6-5bn2d"]
Sep 17 14:18:30.634630 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:30.634601    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b52cd94-d687-482c-92ed-45f29db801ec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-96cfc6-5bn2d\" (UID: \"9b52cd94-d687-482c-92ed-45f29db801ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-96cfc6-5bn2d"
Sep 17 14:18:30.735782 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:30.735692    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b52cd94-d687-482c-92ed-45f29db801ec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-96cfc6-5bn2d\" (UID: \"9b52cd94-d687-482c-92ed-45f29db801ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-96cfc6-5bn2d"
Sep 17 14:18:30.736325 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:30.736298    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b52cd94-d687-482c-92ed-45f29db801ec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-96cfc6-5bn2d\" (UID: \"9b52cd94-d687-482c-92ed-45f29db801ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-96cfc6-5bn2d"
Sep 17 14:18:30.807830 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:30.807802    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-96cfc6-5bn2d"
Sep 17 14:18:30.861640 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:30.861614    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7774-e0678b"]
Sep 17 14:18:30.867088 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:30.866869    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7774-e0678b"]
Sep 17 14:18:30.986571 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:30.986494    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7774-67af63"]
Sep 17 14:18:30.987297 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:30.987277    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef7db703-e60c-4ba4-97cf-6865c11e7841" containerName="test-curl-7774-e0678b"
Sep 17 14:18:30.987344 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:30.987303    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef7db703-e60c-4ba4-97cf-6865c11e7841" containerName="test-curl-7774-e0678b"
Sep 17 14:18:30.987441 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:30.987428    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef7db703-e60c-4ba4-97cf-6865c11e7841" containerName="test-curl-7774-e0678b"
Sep 17 14:18:30.992300 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:30.992271    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7774-67af63"
Sep 17 14:18:30.998081 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:30.998054    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7774-67af63"]
Sep 17 14:18:31.091184 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:31.091158    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-96cfc6-5bn2d"]
Sep 17 14:18:31.092077 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:18:31.092051    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b52cd94_d687_482c_92ed_45f29db801ec.slice/crio-833212b281843d275870150ab8c9665d05465425c3eab5e1885c76b73216f9be WatchSource:0}: Error finding container 833212b281843d275870150ab8c9665d05465425c3eab5e1885c76b73216f9be: Status 404 returned error can't find the container with id 833212b281843d275870150ab8c9665d05465425c3eab5e1885c76b73216f9be
Sep 17 14:18:31.140205 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:31.140171    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z2qs\" (UniqueName: \"kubernetes.io/projected/e9d79de5-f9b3-439d-9836-c7d019ca0e8b-kube-api-access-9z2qs\") pod \"test-curl-7774-67af63\" (UID: \"e9d79de5-f9b3-439d-9836-c7d019ca0e8b\") " pod="opendatahub/test-curl-7774-67af63"
Sep 17 14:18:31.165753 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:31.165721    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:31.241036 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:31.240953    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z2qs\" (UniqueName: \"kubernetes.io/projected/e9d79de5-f9b3-439d-9836-c7d019ca0e8b-kube-api-access-9z2qs\") pod \"test-curl-7774-67af63\" (UID: \"e9d79de5-f9b3-439d-9836-c7d019ca0e8b\") " pod="opendatahub/test-curl-7774-67af63"
Sep 17 14:18:31.242701 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:31.242675    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-96cfc6-5bn2d" event={"ID":"9b52cd94-d687-482c-92ed-45f29db801ec","Type":"ContainerStarted","Data":"833212b281843d275870150ab8c9665d05465425c3eab5e1885c76b73216f9be"}
Sep 17 14:18:31.242913 ip-10-0-129-158 kubenswrapper[2560]: E0917 14:18:31.242893    2560 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7774-e0678b" container="test-curl-7774-e0678b"
Sep 17 14:18:31.250004 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:31.249986    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z2qs\" (UniqueName: \"kubernetes.io/projected/e9d79de5-f9b3-439d-9836-c7d019ca0e8b-kube-api-access-9z2qs\") pod \"test-curl-7774-67af63\" (UID: \"e9d79de5-f9b3-439d-9836-c7d019ca0e8b\") " pod="opendatahub/test-curl-7774-67af63"
Sep 17 14:18:31.304601 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:31.304577    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7774-67af63"
Sep 17 14:18:31.585509 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:31.585478    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7774-67af63"]
Sep 17 14:18:31.585664 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:18:31.585637    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode9d79de5_f9b3_439d_9836_c7d019ca0e8b.slice/crio-09d36d0d38ca6a524fc4b353edb7d87ec2ffef08901339a59d7c32a2adcab686 WatchSource:0}: Error finding container 09d36d0d38ca6a524fc4b353edb7d87ec2ffef08901339a59d7c32a2adcab686: Status 404 returned error can't find the container with id 09d36d0d38ca6a524fc4b353edb7d87ec2ffef08901339a59d7c32a2adcab686
Sep 17 14:18:31.716232 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:31.716199    2560 scope.go:117] "RemoveContainer" containerID="2b4ce32e0a2459468e224814337fb3ad52a8aee8b212a4a34b6f6915e2a6090e"
Sep 17 14:18:32.020436 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:32.020344    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:32.257192 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:32.257159    2560 generic.go:356] "Generic (PLEG): container finished" podID="9b52cd94-d687-482c-92ed-45f29db801ec" containerID="e475bad21a307163f58afbc78700721c1e8c68e471c606b0218df85bdd1ee0ad" exitCode=0
Sep 17 14:18:32.257678 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:32.257248    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-96cfc6-5bn2d" event={"ID":"9b52cd94-d687-482c-92ed-45f29db801ec","Type":"ContainerDied","Data":"e475bad21a307163f58afbc78700721c1e8c68e471c606b0218df85bdd1ee0ad"}
Sep 17 14:18:32.259476 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:32.259441    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7774-67af63" event={"ID":"e9d79de5-f9b3-439d-9836-c7d019ca0e8b","Type":"ContainerStarted","Data":"4965897d01d3f67661443a6c5b658959c48e3c3227e25935ea1b6624dacc6f16"}
Sep 17 14:18:32.259599 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:32.259487    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7774-67af63" event={"ID":"e9d79de5-f9b3-439d-9836-c7d019ca0e8b","Type":"ContainerStarted","Data":"09d36d0d38ca6a524fc4b353edb7d87ec2ffef08901339a59d7c32a2adcab686"}
Sep 17 14:18:32.261074 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:32.261055    2560 generic.go:356] "Generic (PLEG): container finished" podID="c66fc21d-4519-4293-bc1f-4e9ae8fe2564" containerID="9880e4126cffa56d0bbb6f0975de61ee8898a364be70441398c8c624ce64a287" exitCode=0
Sep 17 14:18:32.261158 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:32.261127    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt" event={"ID":"c66fc21d-4519-4293-bc1f-4e9ae8fe2564","Type":"ContainerDied","Data":"9880e4126cffa56d0bbb6f0975de61ee8898a364be70441398c8c624ce64a287"}
Sep 17 14:18:32.261212 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:32.261197    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7774-e0678b" podUID="ef7db703-e60c-4ba4-97cf-6865c11e7841" containerName="test-curl-7774-e0678b" containerID="cri-o://2bd063a43fa40c7e1bca62adec4749518f040698b7d7a8e19693bf904a9aa4d2" gracePeriod=2
Sep 17 14:18:32.261752 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:32.261523    2560 scope.go:117] "RemoveContainer" containerID="9880e4126cffa56d0bbb6f0975de61ee8898a364be70441398c8c624ce64a287"
Sep 17 14:18:32.262635 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:32.262607    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:18:32.262687 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:32.262664    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:18:32.262723 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:32.262694    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:18:32.299078 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:32.299017    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7774-67af63" podStartSLOduration=2.298999549 podStartE2EDuration="2.298999549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:18:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:18:32.297810762 +0000 UTC m=+1381.239954410" watchObservedRunningTime="2026-09-17 14:18:32.298999549 +0000 UTC m=+1381.241143194"
Sep 17 14:18:32.859771 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:32.859737    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:32.912532 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:32.912491    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7774-67af63"]
Sep 17 14:18:32.917073 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:32.916948    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7774-67af63"]
Sep 17 14:18:33.027728 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:33.027650    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-ebc24f/noten-model-ebc24f-kserve-69d64dd88c-cxqkj"]
Sep 17 14:18:33.028151 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:33.028135    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9d79de5-f9b3-439d-9836-c7d019ca0e8b" containerName="test-curl-7774-67af63"
Sep 17 14:18:33.028217 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:33.028153    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9d79de5-f9b3-439d-9836-c7d019ca0e8b" containerName="test-curl-7774-67af63"
Sep 17 14:18:33.028279 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:33.028268    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9d79de5-f9b3-439d-9836-c7d019ca0e8b" containerName="test-curl-7774-67af63"
Sep 17 14:18:33.031600 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:33.031578    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-ebc24f/noten-model-ebc24f-kserve-69d64dd88c-cxqkj"
Sep 17 14:18:33.034876 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:33.034851    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-ebc24f\"/\"default-dockercfg-l2578\""
Sep 17 14:18:33.035045 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:33.034851    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-ebc24f\"/\"noten-model-ebc24f-kserve-self-signed-certs\""
Sep 17 14:18:33.035045 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:33.034894    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-ebc24f\"/\"kube-root-ca.crt\""
Sep 17 14:18:33.035045 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:33.034855    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-ebc24f\"/\"openshift-service-ca.crt\""
Sep 17 14:18:33.040432 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:33.040372    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7774-791659"]
Sep 17 14:18:33.045606 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:33.045201    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-ebc24f/noten-model-ebc24f-kserve-69d64dd88c-cxqkj"]
Sep 17 14:18:33.045606 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:33.045292    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7774-791659"
Sep 17 14:18:33.049591 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:33.049567    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7774-791659"]
Sep 17 14:18:33.064735 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:33.064709    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:33.173483 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:33.170079    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zwrd\" (UniqueName: \"kubernetes.io/projected/54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e-kube-api-access-8zwrd\") pod \"noten-model-ebc24f-kserve-69d64dd88c-cxqkj\" (UID: \"54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e\") " pod="ai-tenant-e2e-noten-ebc24f/noten-model-ebc24f-kserve-69d64dd88c-cxqkj"
Sep 17 14:18:33.173483 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:33.170184    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e-tls-certs\") pod \"noten-model-ebc24f-kserve-69d64dd88c-cxqkj\" (UID: \"54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e\") " pod="ai-tenant-e2e-noten-ebc24f/noten-model-ebc24f-kserve-69d64dd88c-cxqkj"
Sep 17 14:18:33.173483 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:33.170212    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p276d\" (UniqueName: \"kubernetes.io/projected/e701f92e-fe5e-4276-b15d-3db048a0ed8d-kube-api-access-p276d\") pod \"test-curl-7774-791659\" (UID: \"e701f92e-fe5e-4276-b15d-3db048a0ed8d\") " pod="opendatahub/test-curl-7774-791659"
Sep 17 14:18:33.173483 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:33.170266    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e-home\") pod \"noten-model-ebc24f-kserve-69d64dd88c-cxqkj\" (UID: \"54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e\") " pod="ai-tenant-e2e-noten-ebc24f/noten-model-ebc24f-kserve-69d64dd88c-cxqkj"
Sep 17 14:18:33.173483 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:33.170337    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e-dshm\") pod \"noten-model-ebc24f-kserve-69d64dd88c-cxqkj\" (UID: \"54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e\") " pod="ai-tenant-e2e-noten-ebc24f/noten-model-ebc24f-kserve-69d64dd88c-cxqkj"
Sep 17 14:18:33.173483 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:33.170397    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e-model-cache\") pod \"noten-model-ebc24f-kserve-69d64dd88c-cxqkj\" (UID: \"54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e\") " pod="ai-tenant-e2e-noten-ebc24f/noten-model-ebc24f-kserve-69d64dd88c-cxqkj"
Sep 17 14:18:33.173483 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:33.170448    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e-tmp-dir\") pod \"noten-model-ebc24f-kserve-69d64dd88c-cxqkj\" (UID: \"54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e\") " pod="ai-tenant-e2e-noten-ebc24f/noten-model-ebc24f-kserve-69d64dd88c-cxqkj"
Sep 17 14:18:33.271522 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:33.271484    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e-tls-certs\") pod \"noten-model-ebc24f-kserve-69d64dd88c-cxqkj\" (UID: \"54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e\") " pod="ai-tenant-e2e-noten-ebc24f/noten-model-ebc24f-kserve-69d64dd88c-cxqkj"
Sep 17 14:18:33.272004 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:33.271531    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p276d\" (UniqueName: \"kubernetes.io/projected/e701f92e-fe5e-4276-b15d-3db048a0ed8d-kube-api-access-p276d\") pod \"test-curl-7774-791659\" (UID: \"e701f92e-fe5e-4276-b15d-3db048a0ed8d\") " pod="opendatahub/test-curl-7774-791659"
Sep 17 14:18:33.272004 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:33.271589    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e-home\") pod \"noten-model-ebc24f-kserve-69d64dd88c-cxqkj\" (UID: \"54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e\") " pod="ai-tenant-e2e-noten-ebc24f/noten-model-ebc24f-kserve-69d64dd88c-cxqkj"
Sep 17 14:18:33.272004 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:33.271674    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e-dshm\") pod \"noten-model-ebc24f-kserve-69d64dd88c-cxqkj\" (UID: \"54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e\") " pod="ai-tenant-e2e-noten-ebc24f/noten-model-ebc24f-kserve-69d64dd88c-cxqkj"
Sep 17 14:18:33.272004 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:33.271740    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e-model-cache\") pod \"noten-model-ebc24f-kserve-69d64dd88c-cxqkj\" (UID: \"54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e\") " pod="ai-tenant-e2e-noten-ebc24f/noten-model-ebc24f-kserve-69d64dd88c-cxqkj"
Sep 17 14:18:33.272004 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:33.271789    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e-tmp-dir\") pod \"noten-model-ebc24f-kserve-69d64dd88c-cxqkj\" (UID: \"54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e\") " pod="ai-tenant-e2e-noten-ebc24f/noten-model-ebc24f-kserve-69d64dd88c-cxqkj"
Sep 17 14:18:33.272004 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:33.271847    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zwrd\" (UniqueName: \"kubernetes.io/projected/54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e-kube-api-access-8zwrd\") pod \"noten-model-ebc24f-kserve-69d64dd88c-cxqkj\" (UID: \"54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e\") " pod="ai-tenant-e2e-noten-ebc24f/noten-model-ebc24f-kserve-69d64dd88c-cxqkj"
Sep 17 14:18:33.275612 ip-10-0-129-158 kubenswrapper[2560]: E0917 14:18:33.275582    2560 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7774-67af63" container="test-curl-7774-67af63"
Sep 17 14:18:33.275847 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:33.275820    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt" event={"ID":"c66fc21d-4519-4293-bc1f-4e9ae8fe2564","Type":"ContainerStarted","Data":"921815787c1cbc3457545392ec720572c154159e9331f33a9ef283f5be16c858"}
Sep 17 14:18:33.276274 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:33.276247    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e-home\") pod \"noten-model-ebc24f-kserve-69d64dd88c-cxqkj\" (UID: \"54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e\") " pod="ai-tenant-e2e-noten-ebc24f/noten-model-ebc24f-kserve-69d64dd88c-cxqkj"
Sep 17 14:18:33.280520 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:33.277920    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e-tmp-dir\") pod \"noten-model-ebc24f-kserve-69d64dd88c-cxqkj\" (UID: \"54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e\") " pod="ai-tenant-e2e-noten-ebc24f/noten-model-ebc24f-kserve-69d64dd88c-cxqkj"
Sep 17 14:18:33.280520 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:33.278178    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e-model-cache\") pod \"noten-model-ebc24f-kserve-69d64dd88c-cxqkj\" (UID: \"54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e\") " pod="ai-tenant-e2e-noten-ebc24f/noten-model-ebc24f-kserve-69d64dd88c-cxqkj"
Sep 17 14:18:33.280520 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:33.278684    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e-tls-certs\") pod \"noten-model-ebc24f-kserve-69d64dd88c-cxqkj\" (UID: \"54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e\") " pod="ai-tenant-e2e-noten-ebc24f/noten-model-ebc24f-kserve-69d64dd88c-cxqkj"
Sep 17 14:18:33.286713 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:33.286673    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e-dshm\") pod \"noten-model-ebc24f-kserve-69d64dd88c-cxqkj\" (UID: \"54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e\") " pod="ai-tenant-e2e-noten-ebc24f/noten-model-ebc24f-kserve-69d64dd88c-cxqkj"
Sep 17 14:18:33.288408 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:33.288361    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zwrd\" (UniqueName: \"kubernetes.io/projected/54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e-kube-api-access-8zwrd\") pod \"noten-model-ebc24f-kserve-69d64dd88c-cxqkj\" (UID: \"54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e\") " pod="ai-tenant-e2e-noten-ebc24f/noten-model-ebc24f-kserve-69d64dd88c-cxqkj"
Sep 17 14:18:33.288838 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:33.288816    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p276d\" (UniqueName: \"kubernetes.io/projected/e701f92e-fe5e-4276-b15d-3db048a0ed8d-kube-api-access-p276d\") pod \"test-curl-7774-791659\" (UID: \"e701f92e-fe5e-4276-b15d-3db048a0ed8d\") " pod="opendatahub/test-curl-7774-791659"
Sep 17 14:18:33.342785 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:33.342758    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-ebc24f/noten-model-ebc24f-kserve-69d64dd88c-cxqkj"
Sep 17 14:18:33.382762 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:33.371595    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7774-791659"
Sep 17 14:18:33.439074 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:33.439030    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-96cfc6-5bn2d"
Sep 17 14:18:33.593328 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:33.593295    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b52cd94-d687-482c-92ed-45f29db801ec-maas-api-service-ca\") pod \"9b52cd94-d687-482c-92ed-45f29db801ec\" (UID: \"9b52cd94-d687-482c-92ed-45f29db801ec\") "
Sep 17 14:18:33.593688 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:33.593662    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b52cd94-d687-482c-92ed-45f29db801ec-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9b52cd94-d687-482c-92ed-45f29db801ec" (UID: "9b52cd94-d687-482c-92ed-45f29db801ec"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:18:33.680364 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:33.680325    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-ebc24f/noten-model-ebc24f-kserve-69d64dd88c-cxqkj"]
Sep 17 14:18:33.694266 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:33.694246    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b52cd94-d687-482c-92ed-45f29db801ec-maas-api-service-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:18:33.713061 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:33.713038    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7774-791659"]
Sep 17 14:18:33.714616 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:18:33.714585    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode701f92e_fe5e_4276_b15d_3db048a0ed8d.slice/crio-b48343bb17de07219622dd910d0307194989d0bb9bf6276b3fc0011140d45ff6 WatchSource:0}: Error finding container b48343bb17de07219622dd910d0307194989d0bb9bf6276b3fc0011140d45ff6: Status 404 returned error can't find the container with id b48343bb17de07219622dd910d0307194989d0bb9bf6276b3fc0011140d45ff6
Sep 17 14:18:34.066350 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:34.066277    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:34.099585 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:34.099558    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt"
Sep 17 14:18:34.099788 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:34.099769    2560 patch_prober.go:28] interesting pod/e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Sep 17 14:18:34.099834 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:34.099816    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt" podUID="c66fc21d-4519-4293-bc1f-4e9ae8fe2564" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Sep 17 14:18:34.289170 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:34.289139    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7774-791659" event={"ID":"e701f92e-fe5e-4276-b15d-3db048a0ed8d","Type":"ContainerStarted","Data":"b9ed8de6c8c627105b8db78f7c1debf5d343859c3ed6cbf5e83ace28fa3b9398"}
Sep 17 14:18:34.289554 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:34.289185    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7774-791659" event={"ID":"e701f92e-fe5e-4276-b15d-3db048a0ed8d","Type":"ContainerStarted","Data":"b48343bb17de07219622dd910d0307194989d0bb9bf6276b3fc0011140d45ff6"}
Sep 17 14:18:34.290536 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:34.290516    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-96cfc6-5bn2d"
Sep 17 14:18:34.290632 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:34.290552    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-96cfc6-5bn2d" event={"ID":"9b52cd94-d687-482c-92ed-45f29db801ec","Type":"ContainerDied","Data":"833212b281843d275870150ab8c9665d05465425c3eab5e1885c76b73216f9be"}
Sep 17 14:18:34.290632 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:34.290598    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="833212b281843d275870150ab8c9665d05465425c3eab5e1885c76b73216f9be"
Sep 17 14:18:34.292550 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:34.292495    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-ebc24f/noten-model-ebc24f-kserve-69d64dd88c-cxqkj" event={"ID":"54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e","Type":"ContainerStarted","Data":"2dfed35f65ab3bd51630b3269ddfb03dcc53e1438a434605a68adc2adb94bb08"}
Sep 17 14:18:34.292550 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:34.292525    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-ebc24f/noten-model-ebc24f-kserve-69d64dd88c-cxqkj" event={"ID":"54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e","Type":"ContainerStarted","Data":"647c4fc778e71ad3337312132268e348165d7a71c9a02d7392cd0eb3fdaea4ad"}
Sep 17 14:18:34.292880 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:34.292854    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7774-67af63" podUID="e9d79de5-f9b3-439d-9836-c7d019ca0e8b" containerName="test-curl-7774-67af63" containerID="cri-o://4965897d01d3f67661443a6c5b658959c48e3c3227e25935ea1b6624dacc6f16" gracePeriod=2
Sep 17 14:18:34.305204 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:34.305167    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7774-791659" podStartSLOduration=1.305154272 podStartE2EDuration="1.305154272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:18:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:18:34.304010744 +0000 UTC m=+1383.246154391" watchObservedRunningTime="2026-09-17 14:18:34.305154272 +0000 UTC m=+1383.247297912"
Sep 17 14:18:34.324620 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:34.324584    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-ebc24f/noten-model-ebc24f-kserve-69d64dd88c-cxqkj" podStartSLOduration=1.324571717 podStartE2EDuration="1.324571717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:18:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:18:34.32176429 +0000 UTC m=+1383.263907937" watchObservedRunningTime="2026-09-17 14:18:34.324571717 +0000 UTC m=+1383.266715362"
Sep 17 14:18:34.500715 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:34.500672    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7774-e0678b"
Sep 17 14:18:34.602071 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:34.601995    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lf2hk\" (UniqueName: \"kubernetes.io/projected/ef7db703-e60c-4ba4-97cf-6865c11e7841-kube-api-access-lf2hk\") pod \"ef7db703-e60c-4ba4-97cf-6865c11e7841\" (UID: \"ef7db703-e60c-4ba4-97cf-6865c11e7841\") "
Sep 17 14:18:34.604022 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:34.603995    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef7db703-e60c-4ba4-97cf-6865c11e7841-kube-api-access-lf2hk" (OuterVolumeSpecName: "kube-api-access-lf2hk") pod "ef7db703-e60c-4ba4-97cf-6865c11e7841" (UID: "ef7db703-e60c-4ba4-97cf-6865c11e7841"). InnerVolumeSpecName "kube-api-access-lf2hk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:18:34.703207 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:34.703184    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lf2hk\" (UniqueName: \"kubernetes.io/projected/ef7db703-e60c-4ba4-97cf-6865c11e7841-kube-api-access-lf2hk\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:18:34.912780 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:34.912688    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:34.922687 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:34.922662    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7774-791659"]
Sep 17 14:18:34.928253 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:34.928226    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7774-791659"]
Sep 17 14:18:35.099758 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:35.099727    2560 patch_prober.go:28] interesting pod/e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Sep 17 14:18:35.099920 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:35.099782    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt" podUID="c66fc21d-4519-4293-bc1f-4e9ae8fe2564" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Sep 17 14:18:35.300260 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:35.300229    2560 generic.go:356] "Generic (PLEG): container finished" podID="ef7db703-e60c-4ba4-97cf-6865c11e7841" containerID="2bd063a43fa40c7e1bca62adec4749518f040698b7d7a8e19693bf904a9aa4d2" exitCode=137
Sep 17 14:18:35.300713 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:35.300273    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7774-e0678b"
Sep 17 14:18:35.300713 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:35.300338    2560 scope.go:117] "RemoveContainer" containerID="2bd063a43fa40c7e1bca62adec4749518f040698b7d7a8e19693bf904a9aa4d2"
Sep 17 14:18:35.300713 ip-10-0-129-158 kubenswrapper[2560]: E0917 14:18:35.300586    2560 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7774-791659" container="test-curl-7774-791659"
Sep 17 14:18:35.310835 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:35.310818    2560 scope.go:117] "RemoveContainer" containerID="2bd063a43fa40c7e1bca62adec4749518f040698b7d7a8e19693bf904a9aa4d2"
Sep 17 14:18:35.311085 ip-10-0-129-158 kubenswrapper[2560]: E0917 14:18:35.311057    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bd063a43fa40c7e1bca62adec4749518f040698b7d7a8e19693bf904a9aa4d2\": container with ID starting with 2bd063a43fa40c7e1bca62adec4749518f040698b7d7a8e19693bf904a9aa4d2 not found: ID does not exist" containerID="2bd063a43fa40c7e1bca62adec4749518f040698b7d7a8e19693bf904a9aa4d2"
Sep 17 14:18:35.311150 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:35.311095    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bd063a43fa40c7e1bca62adec4749518f040698b7d7a8e19693bf904a9aa4d2"} err="failed to get container status \"2bd063a43fa40c7e1bca62adec4749518f040698b7d7a8e19693bf904a9aa4d2\": rpc error: code = NotFound desc = could not find container \"2bd063a43fa40c7e1bca62adec4749518f040698b7d7a8e19693bf904a9aa4d2\": container with ID starting with 2bd063a43fa40c7e1bca62adec4749518f040698b7d7a8e19693bf904a9aa4d2 not found: ID does not exist"
Sep 17 14:18:35.311199 ip-10-0-129-158 kubenswrapper[2560]: E0917 14:18:35.311166    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7774-e0678b\" is forbidden: User \"system:node:ip-10-0-129-158.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-158.ec2.internal' and this object" podUID="ef7db703-e60c-4ba4-97cf-6865c11e7841" pod="opendatahub/test-curl-7774-e0678b"
Sep 17 14:18:35.576382 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:35.576288    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef7db703-e60c-4ba4-97cf-6865c11e7841" path="/var/lib/kubelet/pods/ef7db703-e60c-4ba4-97cf-6865c11e7841/volumes"
Sep 17 14:18:35.926501 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:35.926402    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:36.099701 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:36.099665    2560 patch_prober.go:28] interesting pod/e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Sep 17 14:18:36.099886 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:36.099748    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt" podUID="c66fc21d-4519-4293-bc1f-4e9ae8fe2564" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Sep 17 14:18:36.122555 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:36.122522    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:36.309195 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:36.309156    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7774-791659" podUID="e701f92e-fe5e-4276-b15d-3db048a0ed8d" containerName="test-curl-7774-791659" containerID="cri-o://b9ed8de6c8c627105b8db78f7c1debf5d343859c3ed6cbf5e83ace28fa3b9398" gracePeriod=2
Sep 17 14:18:36.482807 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:36.482770    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7774-67af63"
Sep 17 14:18:36.519936 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:36.519907    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9z2qs\" (UniqueName: \"kubernetes.io/projected/e9d79de5-f9b3-439d-9836-c7d019ca0e8b-kube-api-access-9z2qs\") pod \"e9d79de5-f9b3-439d-9836-c7d019ca0e8b\" (UID: \"e9d79de5-f9b3-439d-9836-c7d019ca0e8b\") "
Sep 17 14:18:36.521966 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:36.521943    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9d79de5-f9b3-439d-9836-c7d019ca0e8b-kube-api-access-9z2qs" (OuterVolumeSpecName: "kube-api-access-9z2qs") pod "e9d79de5-f9b3-439d-9836-c7d019ca0e8b" (UID: "e9d79de5-f9b3-439d-9836-c7d019ca0e8b"). InnerVolumeSpecName "kube-api-access-9z2qs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:18:36.621476 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:36.621440    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9z2qs\" (UniqueName: \"kubernetes.io/projected/e9d79de5-f9b3-439d-9836-c7d019ca0e8b-kube-api-access-9z2qs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:18:36.898920 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:36.898816    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:37.099601 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:37.099562    2560 patch_prober.go:28] interesting pod/e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Sep 17 14:18:37.099791 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:37.099627    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt" podUID="c66fc21d-4519-4293-bc1f-4e9ae8fe2564" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Sep 17 14:18:37.317386 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:37.317354    2560 generic.go:356] "Generic (PLEG): container finished" podID="e9d79de5-f9b3-439d-9836-c7d019ca0e8b" containerID="4965897d01d3f67661443a6c5b658959c48e3c3227e25935ea1b6624dacc6f16" exitCode=137
Sep 17 14:18:37.317823 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:37.317390    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7774-67af63"
Sep 17 14:18:37.317823 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:37.317441    2560 scope.go:117] "RemoveContainer" containerID="4965897d01d3f67661443a6c5b658959c48e3c3227e25935ea1b6624dacc6f16"
Sep 17 14:18:37.327111 ip-10-0-129-158 kubenswrapper[2560]: E0917 14:18:37.327084    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7774-67af63\" is forbidden: User \"system:node:ip-10-0-129-158.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-158.ec2.internal' and this object" podUID="e9d79de5-f9b3-439d-9836-c7d019ca0e8b" pod="opendatahub/test-curl-7774-67af63"
Sep 17 14:18:37.327300 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:37.327281    2560 scope.go:117] "RemoveContainer" containerID="4965897d01d3f67661443a6c5b658959c48e3c3227e25935ea1b6624dacc6f16"
Sep 17 14:18:37.327621 ip-10-0-129-158 kubenswrapper[2560]: E0917 14:18:37.327598    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4965897d01d3f67661443a6c5b658959c48e3c3227e25935ea1b6624dacc6f16\": container with ID starting with 4965897d01d3f67661443a6c5b658959c48e3c3227e25935ea1b6624dacc6f16 not found: ID does not exist" containerID="4965897d01d3f67661443a6c5b658959c48e3c3227e25935ea1b6624dacc6f16"
Sep 17 14:18:37.327713 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:37.327627    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4965897d01d3f67661443a6c5b658959c48e3c3227e25935ea1b6624dacc6f16"} err="failed to get container status \"4965897d01d3f67661443a6c5b658959c48e3c3227e25935ea1b6624dacc6f16\": rpc error: code = NotFound desc = could not find container \"4965897d01d3f67661443a6c5b658959c48e3c3227e25935ea1b6624dacc6f16\": container with ID starting with 4965897d01d3f67661443a6c5b658959c48e3c3227e25935ea1b6624dacc6f16 not found: ID does not exist"
Sep 17 14:18:37.589686 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:37.589593    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9d79de5-f9b3-439d-9836-c7d019ca0e8b" path="/var/lib/kubelet/pods/e9d79de5-f9b3-439d-9836-c7d019ca0e8b/volumes"
Sep 17 14:18:37.694399 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:37.694359    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:37.984710 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:37.984626    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h"]
Sep 17 14:18:37.985617 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:37.985592    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b52cd94-d687-482c-92ed-45f29db801ec" containerName="revoke-keys"
Sep 17 14:18:37.985756 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:37.985745    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b52cd94-d687-482c-92ed-45f29db801ec" containerName="revoke-keys"
Sep 17 14:18:37.985875 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:37.985864    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e701f92e-fe5e-4276-b15d-3db048a0ed8d" containerName="test-curl-7774-791659"
Sep 17 14:18:37.985949 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:37.985940    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e701f92e-fe5e-4276-b15d-3db048a0ed8d" containerName="test-curl-7774-791659"
Sep 17 14:18:37.986158 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:37.986146    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e701f92e-fe5e-4276-b15d-3db048a0ed8d" containerName="test-curl-7774-791659"
Sep 17 14:18:37.986245 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:37.986235    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b52cd94-d687-482c-92ed-45f29db801ec" containerName="revoke-keys"
Sep 17 14:18:37.991369 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:37.991348    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h"]
Sep 17 14:18:37.991481 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:37.991441    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h"
Sep 17 14:18:37.993551 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:37.993525    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-5cdd59e7-openshift-default-dockercfg-wf44r\""
Sep 17 14:18:38.009438 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:38.009403    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:38.037753 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:38.037729    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98a693c5-386f-4525-b3e1-7f891ef7d68a-credential-socket\") pod \"e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h\" (UID: \"98a693c5-386f-4525-b3e1-7f891ef7d68a\") " pod="openshift-ingress/e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h"
Sep 17 14:18:38.037865 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:38.037768    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98a693c5-386f-4525-b3e1-7f891ef7d68a-istio-envoy\") pod \"e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h\" (UID: \"98a693c5-386f-4525-b3e1-7f891ef7d68a\") " pod="openshift-ingress/e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h"
Sep 17 14:18:38.037865 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:38.037792    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98a693c5-386f-4525-b3e1-7f891ef7d68a-istio-podinfo\") pod \"e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h\" (UID: \"98a693c5-386f-4525-b3e1-7f891ef7d68a\") " pod="openshift-ingress/e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h"
Sep 17 14:18:38.037938 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:38.037874    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98a693c5-386f-4525-b3e1-7f891ef7d68a-istio-data\") pod \"e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h\" (UID: \"98a693c5-386f-4525-b3e1-7f891ef7d68a\") " pod="openshift-ingress/e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h"
Sep 17 14:18:38.037974 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:38.037936    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98a693c5-386f-4525-b3e1-7f891ef7d68a-workload-certs\") pod \"e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h\" (UID: \"98a693c5-386f-4525-b3e1-7f891ef7d68a\") " pod="openshift-ingress/e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h"
Sep 17 14:18:38.038016 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:38.038000    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98a693c5-386f-4525-b3e1-7f891ef7d68a-workload-socket\") pod \"e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h\" (UID: \"98a693c5-386f-4525-b3e1-7f891ef7d68a\") " pod="openshift-ingress/e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h"
Sep 17 14:18:38.038053 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:38.038038    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvqbr\" (UniqueName: \"kubernetes.io/projected/98a693c5-386f-4525-b3e1-7f891ef7d68a-kube-api-access-pvqbr\") pod \"e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h\" (UID: \"98a693c5-386f-4525-b3e1-7f891ef7d68a\") " pod="openshift-ingress/e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h"
Sep 17 14:18:38.038101 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:38.038086    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98a693c5-386f-4525-b3e1-7f891ef7d68a-istiod-ca-cert\") pod \"e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h\" (UID: \"98a693c5-386f-4525-b3e1-7f891ef7d68a\") " pod="openshift-ingress/e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h"
Sep 17 14:18:38.038134 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:38.038117    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98a693c5-386f-4525-b3e1-7f891ef7d68a-istio-token\") pod \"e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h\" (UID: \"98a693c5-386f-4525-b3e1-7f891ef7d68a\") " pod="openshift-ingress/e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h"
Sep 17 14:18:38.100668 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:38.100632    2560 patch_prober.go:28] interesting pod/e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Sep 17 14:18:38.100993 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:38.100958    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt" podUID="c66fc21d-4519-4293-bc1f-4e9ae8fe2564" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Sep 17 14:18:38.139381 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:38.139348    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98a693c5-386f-4525-b3e1-7f891ef7d68a-istio-envoy\") pod \"e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h\" (UID: \"98a693c5-386f-4525-b3e1-7f891ef7d68a\") " pod="openshift-ingress/e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h"
Sep 17 14:18:38.139547 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:38.139402    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98a693c5-386f-4525-b3e1-7f891ef7d68a-istio-podinfo\") pod \"e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h\" (UID: \"98a693c5-386f-4525-b3e1-7f891ef7d68a\") " pod="openshift-ingress/e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h"
Sep 17 14:18:38.139547 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:38.139430    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98a693c5-386f-4525-b3e1-7f891ef7d68a-istio-data\") pod \"e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h\" (UID: \"98a693c5-386f-4525-b3e1-7f891ef7d68a\") " pod="openshift-ingress/e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h"
Sep 17 14:18:38.139547 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:38.139502    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98a693c5-386f-4525-b3e1-7f891ef7d68a-workload-certs\") pod \"e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h\" (UID: \"98a693c5-386f-4525-b3e1-7f891ef7d68a\") " pod="openshift-ingress/e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h"
Sep 17 14:18:38.139692 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:38.139563    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98a693c5-386f-4525-b3e1-7f891ef7d68a-workload-socket\") pod \"e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h\" (UID: \"98a693c5-386f-4525-b3e1-7f891ef7d68a\") " pod="openshift-ingress/e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h"
Sep 17 14:18:38.139692 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:38.139602    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvqbr\" (UniqueName: \"kubernetes.io/projected/98a693c5-386f-4525-b3e1-7f891ef7d68a-kube-api-access-pvqbr\") pod \"e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h\" (UID: \"98a693c5-386f-4525-b3e1-7f891ef7d68a\") " pod="openshift-ingress/e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h"
Sep 17 14:18:38.139692 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:38.139644    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98a693c5-386f-4525-b3e1-7f891ef7d68a-istiod-ca-cert\") pod \"e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h\" (UID: \"98a693c5-386f-4525-b3e1-7f891ef7d68a\") " pod="openshift-ingress/e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h"
Sep 17 14:18:38.139692 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:38.139678    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98a693c5-386f-4525-b3e1-7f891ef7d68a-istio-token\") pod \"e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h\" (UID: \"98a693c5-386f-4525-b3e1-7f891ef7d68a\") " pod="openshift-ingress/e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h"
Sep 17 14:18:38.139913 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:38.139767    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98a693c5-386f-4525-b3e1-7f891ef7d68a-credential-socket\") pod \"e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h\" (UID: \"98a693c5-386f-4525-b3e1-7f891ef7d68a\") " pod="openshift-ingress/e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h"
Sep 17 14:18:38.139913 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:38.139898    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98a693c5-386f-4525-b3e1-7f891ef7d68a-istio-data\") pod \"e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h\" (UID: \"98a693c5-386f-4525-b3e1-7f891ef7d68a\") " pod="openshift-ingress/e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h"
Sep 17 14:18:38.140022 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:38.139931    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98a693c5-386f-4525-b3e1-7f891ef7d68a-workload-socket\") pod \"e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h\" (UID: \"98a693c5-386f-4525-b3e1-7f891ef7d68a\") " pod="openshift-ingress/e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h"
Sep 17 14:18:38.140022 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:38.139957    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98a693c5-386f-4525-b3e1-7f891ef7d68a-workload-certs\") pod \"e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h\" (UID: \"98a693c5-386f-4525-b3e1-7f891ef7d68a\") " pod="openshift-ingress/e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h"
Sep 17 14:18:38.140260 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:38.140235    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98a693c5-386f-4525-b3e1-7f891ef7d68a-credential-socket\") pod \"e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h\" (UID: \"98a693c5-386f-4525-b3e1-7f891ef7d68a\") " pod="openshift-ingress/e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h"
Sep 17 14:18:38.140527 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:38.140509    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98a693c5-386f-4525-b3e1-7f891ef7d68a-istiod-ca-cert\") pod \"e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h\" (UID: \"98a693c5-386f-4525-b3e1-7f891ef7d68a\") " pod="openshift-ingress/e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h"
Sep 17 14:18:38.141792 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:38.141772    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98a693c5-386f-4525-b3e1-7f891ef7d68a-istio-envoy\") pod \"e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h\" (UID: \"98a693c5-386f-4525-b3e1-7f891ef7d68a\") " pod="openshift-ingress/e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h"
Sep 17 14:18:38.142100 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:38.142078    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98a693c5-386f-4525-b3e1-7f891ef7d68a-istio-podinfo\") pod \"e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h\" (UID: \"98a693c5-386f-4525-b3e1-7f891ef7d68a\") " pod="openshift-ingress/e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h"
Sep 17 14:18:38.147795 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:38.147773    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98a693c5-386f-4525-b3e1-7f891ef7d68a-istio-token\") pod \"e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h\" (UID: \"98a693c5-386f-4525-b3e1-7f891ef7d68a\") " pod="openshift-ingress/e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h"
Sep 17 14:18:38.148003 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:38.147984    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvqbr\" (UniqueName: \"kubernetes.io/projected/98a693c5-386f-4525-b3e1-7f891ef7d68a-kube-api-access-pvqbr\") pod \"e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h\" (UID: \"98a693c5-386f-4525-b3e1-7f891ef7d68a\") " pod="openshift-ingress/e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h"
Sep 17 14:18:38.302297 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:38.302263    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h"
Sep 17 14:18:38.382521 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:38.375380    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:38.543542 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:38.543520    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7774-791659"
Sep 17 14:18:38.594641 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:38.594617    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h"]
Sep 17 14:18:38.596690 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:18:38.596659    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98a693c5_386f_4525_b3e1_7f891ef7d68a.slice/crio-37f877121aa145ac3f5fcf8f8aad779e2ec4c37713d8470519f22d4b4b042b08 WatchSource:0}: Error finding container 37f877121aa145ac3f5fcf8f8aad779e2ec4c37713d8470519f22d4b4b042b08: Status 404 returned error can't find the container with id 37f877121aa145ac3f5fcf8f8aad779e2ec4c37713d8470519f22d4b4b042b08
Sep 17 14:18:38.599249 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:38.599219    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:18:38.599315 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:38.599285    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:18:38.599355 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:38.599315    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:18:38.644373 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:38.644347    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p276d\" (UniqueName: \"kubernetes.io/projected/e701f92e-fe5e-4276-b15d-3db048a0ed8d-kube-api-access-p276d\") pod \"e701f92e-fe5e-4276-b15d-3db048a0ed8d\" (UID: \"e701f92e-fe5e-4276-b15d-3db048a0ed8d\") "
Sep 17 14:18:38.646256 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:38.646230    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e701f92e-fe5e-4276-b15d-3db048a0ed8d-kube-api-access-p276d" (OuterVolumeSpecName: "kube-api-access-p276d") pod "e701f92e-fe5e-4276-b15d-3db048a0ed8d" (UID: "e701f92e-fe5e-4276-b15d-3db048a0ed8d"). InnerVolumeSpecName "kube-api-access-p276d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:18:38.746607 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:38.746557    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p276d\" (UniqueName: \"kubernetes.io/projected/e701f92e-fe5e-4276-b15d-3db048a0ed8d-kube-api-access-p276d\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:18:39.103824 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:39.103792    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt"
Sep 17 14:18:39.104080 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:39.104058    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt"
Sep 17 14:18:39.104682 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:39.104664    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt"
Sep 17 14:18:39.336040 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:39.336006    2560 generic.go:356] "Generic (PLEG): container finished" podID="e701f92e-fe5e-4276-b15d-3db048a0ed8d" containerID="b9ed8de6c8c627105b8db78f7c1debf5d343859c3ed6cbf5e83ace28fa3b9398" exitCode=137
Sep 17 14:18:39.336211 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:39.336056    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7774-791659"
Sep 17 14:18:39.336211 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:39.336096    2560 scope.go:117] "RemoveContainer" containerID="b9ed8de6c8c627105b8db78f7c1debf5d343859c3ed6cbf5e83ace28fa3b9398"
Sep 17 14:18:39.338095 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:39.338070    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h" event={"ID":"98a693c5-386f-4525-b3e1-7f891ef7d68a","Type":"ContainerStarted","Data":"6cb7b2daac5fd38d7fa3dda92140b61a7e71ce7da14c7accd5cab2dd3bd4b32b"}
Sep 17 14:18:39.338233 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:39.338129    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h" event={"ID":"98a693c5-386f-4525-b3e1-7f891ef7d68a","Type":"ContainerStarted","Data":"37f877121aa145ac3f5fcf8f8aad779e2ec4c37713d8470519f22d4b4b042b08"}
Sep 17 14:18:39.346185 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:39.346168    2560 scope.go:117] "RemoveContainer" containerID="b9ed8de6c8c627105b8db78f7c1debf5d343859c3ed6cbf5e83ace28fa3b9398"
Sep 17 14:18:39.346448 ip-10-0-129-158 kubenswrapper[2560]: E0917 14:18:39.346427    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9ed8de6c8c627105b8db78f7c1debf5d343859c3ed6cbf5e83ace28fa3b9398\": container with ID starting with b9ed8de6c8c627105b8db78f7c1debf5d343859c3ed6cbf5e83ace28fa3b9398 not found: ID does not exist" containerID="b9ed8de6c8c627105b8db78f7c1debf5d343859c3ed6cbf5e83ace28fa3b9398"
Sep 17 14:18:39.346543 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:39.346482    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9ed8de6c8c627105b8db78f7c1debf5d343859c3ed6cbf5e83ace28fa3b9398"} err="failed to get container status \"b9ed8de6c8c627105b8db78f7c1debf5d343859c3ed6cbf5e83ace28fa3b9398\": rpc error: code = NotFound desc = could not find container \"b9ed8de6c8c627105b8db78f7c1debf5d343859c3ed6cbf5e83ace28fa3b9398\": container with ID starting with b9ed8de6c8c627105b8db78f7c1debf5d343859c3ed6cbf5e83ace28fa3b9398 not found: ID does not exist"
Sep 17 14:18:39.361547 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:39.361443    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h" podStartSLOduration=2.361424759 podStartE2EDuration="2.361424759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:18:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:18:39.360919826 +0000 UTC m=+1388.303063472" watchObservedRunningTime="2026-09-17 14:18:39.361424759 +0000 UTC m=+1388.303568397"
Sep 17 14:18:39.362754 ip-10-0-129-158 kubenswrapper[2560]: E0917 14:18:39.362726    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7774-791659\" is forbidden: User \"system:node:ip-10-0-129-158.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-158.ec2.internal' and this object" podUID="e701f92e-fe5e-4276-b15d-3db048a0ed8d" pod="opendatahub/test-curl-7774-791659"
Sep 17 14:18:39.570142 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:39.570109    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e701f92e-fe5e-4276-b15d-3db048a0ed8d" path="/var/lib/kubelet/pods/e701f92e-fe5e-4276-b15d-3db048a0ed8d/volumes"
Sep 17 14:18:40.302473 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:40.302443    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h"
Sep 17 14:18:40.307294 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:40.307266    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h"
Sep 17 14:18:40.345938 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:40.345915    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h"
Sep 17 14:18:40.346779 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:40.346762    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h"
Sep 17 14:18:41.332890 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:41.332846    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:41.437584 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:41.437553    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:41.542774 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:41.542739    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:42.452232 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:42.452201    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:43.343123 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:43.343089    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-ebc24f/noten-model-ebc24f-kserve-69d64dd88c-cxqkj"
Sep 17 14:18:43.343336 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:43.343313    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-ebc24f/noten-model-ebc24f-kserve-69d64dd88c-cxqkj"
Sep 17 14:18:43.355828 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:43.355808    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-ebc24f/noten-model-ebc24f-kserve-69d64dd88c-cxqkj"
Sep 17 14:18:43.366659 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:43.366623    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:43.378136 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:43.378111    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-ebc24f/noten-model-ebc24f-kserve-69d64dd88c-cxqkj"
Sep 17 14:18:44.235035 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:44.235000    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:44.972134 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:44.972097    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-96cfc6-29827575-4t4k8"]
Sep 17 14:18:44.978129 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:44.978103    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-96cfc6-29827575-4t4k8"]
Sep 17 14:18:44.979344 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:44.979320    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:45.078834 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:45.078803    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:45.183407 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:45.183372    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:45.283032 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:45.282938    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:45.298778 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:45.298748    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-96cfc6-5bn2d"]
Sep 17 14:18:45.304378 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:45.304353    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-96cfc6-5bn2d"]
Sep 17 14:18:45.423266 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:45.423235    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:45.570627 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:45.570599    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8559995e-b946-4a42-b149-eda43a4c0e5a" path="/var/lib/kubelet/pods/8559995e-b946-4a42-b149-eda43a4c0e5a/volumes"
Sep 17 14:18:45.570929 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:45.570916    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b52cd94-d687-482c-92ed-45f29db801ec" path="/var/lib/kubelet/pods/9b52cd94-d687-482c-92ed-45f29db801ec/volumes"
Sep 17 14:18:45.660950 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:45.660887    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:46.683220 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:46.683177    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:47.665976 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:47.665948    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:47.757446 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:47.757408    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:48.090424 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:48.090369    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:48.199100 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:48.199066    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:48.299198 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:48.299162    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:48.622552 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:48.622514    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:49.427590 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:49.427549    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:50.753730 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:50.753693    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:50.852484 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:50.852434    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:51.073013 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:51.072984    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:52.286768 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:52.286719    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:53.244167 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:53.244121    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:57.571106 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:57.571074    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-fe6ee5/test-model-fe6ee5-kserve-7774b7d84f-rn4bs"]
Sep 17 14:18:57.614205 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:57.614171    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:57.931274 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:57.931192    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:58.223526 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:58.223413    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4"]
Sep 17 14:18:58.223684 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:58.223648    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4" podUID="08fb3245-86c7-4cab-86d9-3b5177ca5101" containerName="istio-proxy" containerID="cri-o://51d5b57fff878dd6b5d34d3dea78ff25bdd429ebecf3a0ebba0952328be5580b" gracePeriod=30
Sep 17 14:18:58.275388 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:58.275345    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:58.574750 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:58.574715    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fe6ee5-t9qj7"]
Sep 17 14:18:58.579741 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:58.579723    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fe6ee5-t9qj7"
Sep 17 14:18:58.583145 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:58.583125    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 14:18:58.583695 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:58.583679    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rp9ww\""
Sep 17 14:18:58.592853 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:58.592831    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fe6ee5-t9qj7"]
Sep 17 14:18:58.649039 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:58.648996    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0aca08da-5a7b-4eda-9a66-10780ccc2da9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-fe6ee5-t9qj7\" (UID: \"0aca08da-5a7b-4eda-9a66-10780ccc2da9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fe6ee5-t9qj7"
Sep 17 14:18:58.749890 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:58.749854    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0aca08da-5a7b-4eda-9a66-10780ccc2da9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-fe6ee5-t9qj7\" (UID: \"0aca08da-5a7b-4eda-9a66-10780ccc2da9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fe6ee5-t9qj7"
Sep 17 14:18:58.750541 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:58.750518    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0aca08da-5a7b-4eda-9a66-10780ccc2da9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-fe6ee5-t9qj7\" (UID: \"0aca08da-5a7b-4eda-9a66-10780ccc2da9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fe6ee5-t9qj7"
Sep 17 14:18:58.830749 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:58.830654    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:58.891749 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:58.891716    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fe6ee5-t9qj7"
Sep 17 14:18:59.027697 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:59.027658    2560 patch_prober.go:28] interesting pod/e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 17 14:18:59.027873 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:59.027728    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4" podUID="08fb3245-86c7-4cab-86d9-3b5177ca5101" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 14:18:59.169536 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:59.169504    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fe6ee5-t9qj7"]
Sep 17 14:18:59.170926 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:18:59.170903    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aca08da_5a7b_4eda_9a66_10780ccc2da9.slice/crio-3c029115569c53286f2ffeaba50ad315bdd11543ffaeb9d51124f2a36bdadb50 WatchSource:0}: Error finding container 3c029115569c53286f2ffeaba50ad315bdd11543ffaeb9d51124f2a36bdadb50: Status 404 returned error can't find the container with id 3c029115569c53286f2ffeaba50ad315bdd11543ffaeb9d51124f2a36bdadb50
Sep 17 14:18:59.502789 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:59.502754    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fe6ee5-t9qj7" event={"ID":"0aca08da-5a7b-4eda-9a66-10780ccc2da9","Type":"ContainerStarted","Data":"fc82eda55820ffdeace6372738ec079690091d4abfd57cefff79a7192d44667d"}
Sep 17 14:18:59.502908 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:59.502814    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fe6ee5-t9qj7" event={"ID":"0aca08da-5a7b-4eda-9a66-10780ccc2da9","Type":"ContainerStarted","Data":"3c029115569c53286f2ffeaba50ad315bdd11543ffaeb9d51124f2a36bdadb50"}
Sep 17 14:18:59.522906 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:59.522852    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fe6ee5-t9qj7" podStartSLOduration=1.5228343739999999 podStartE2EDuration="1.522834374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:18:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:18:59.520409037 +0000 UTC m=+1408.462552685" watchObservedRunningTime="2026-09-17 14:18:59.522834374 +0000 UTC m=+1408.464978021"
Sep 17 14:18:59.728994 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:18:59.728900    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:00.479212 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:00.479177    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:00.511468 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:00.511435    2560 generic.go:356] "Generic (PLEG): container finished" podID="0aca08da-5a7b-4eda-9a66-10780ccc2da9" containerID="fc82eda55820ffdeace6372738ec079690091d4abfd57cefff79a7192d44667d" exitCode=0
Sep 17 14:19:00.511615 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:00.511523    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fe6ee5-t9qj7" event={"ID":"0aca08da-5a7b-4eda-9a66-10780ccc2da9","Type":"ContainerDied","Data":"fc82eda55820ffdeace6372738ec079690091d4abfd57cefff79a7192d44667d"}
Sep 17 14:19:01.321931 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:01.321899    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:01.659292 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:01.659269    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fe6ee5-t9qj7"
Sep 17 14:19:01.783000 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:01.782959    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0aca08da-5a7b-4eda-9a66-10780ccc2da9-maas-api-service-ca\") pod \"0aca08da-5a7b-4eda-9a66-10780ccc2da9\" (UID: \"0aca08da-5a7b-4eda-9a66-10780ccc2da9\") "
Sep 17 14:19:01.783424 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:01.783396    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0aca08da-5a7b-4eda-9a66-10780ccc2da9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0aca08da-5a7b-4eda-9a66-10780ccc2da9" (UID: "0aca08da-5a7b-4eda-9a66-10780ccc2da9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:19:01.884673 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:01.884632    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0aca08da-5a7b-4eda-9a66-10780ccc2da9-maas-api-service-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:19:02.278651 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:02.278610    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:02.526415 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:02.526384    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fe6ee5-t9qj7"
Sep 17 14:19:02.526836 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:02.526423    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fe6ee5-t9qj7" event={"ID":"0aca08da-5a7b-4eda-9a66-10780ccc2da9","Type":"ContainerDied","Data":"3c029115569c53286f2ffeaba50ad315bdd11543ffaeb9d51124f2a36bdadb50"}
Sep 17 14:19:02.526836 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:02.526519    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c029115569c53286f2ffeaba50ad315bdd11543ffaeb9d51124f2a36bdadb50"
Sep 17 14:19:03.228442 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:03.228396    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:03.479322 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:03.479261    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4"
Sep 17 14:19:03.535913 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:03.535878    2560 generic.go:356] "Generic (PLEG): container finished" podID="08fb3245-86c7-4cab-86d9-3b5177ca5101" containerID="51d5b57fff878dd6b5d34d3dea78ff25bdd429ebecf3a0ebba0952328be5580b" exitCode=0
Sep 17 14:19:03.536385 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:03.535928    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4"
Sep 17 14:19:03.536385 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:03.535960    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4" event={"ID":"08fb3245-86c7-4cab-86d9-3b5177ca5101","Type":"ContainerDied","Data":"51d5b57fff878dd6b5d34d3dea78ff25bdd429ebecf3a0ebba0952328be5580b"}
Sep 17 14:19:03.536385 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:03.536024    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4" event={"ID":"08fb3245-86c7-4cab-86d9-3b5177ca5101","Type":"ContainerDied","Data":"15fca760cfbc68055cfe475d208d6e7655d28191e00b381a59b69cc9e8829a4f"}
Sep 17 14:19:03.536385 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:03.536064    2560 scope.go:117] "RemoveContainer" containerID="51d5b57fff878dd6b5d34d3dea78ff25bdd429ebecf3a0ebba0952328be5580b"
Sep 17 14:19:03.546218 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:03.546194    2560 scope.go:117] "RemoveContainer" containerID="3a6f61ffc5af81c362c8f2df3e6e6509b26ef1fe03f530db443d6b30a487fc7f"
Sep 17 14:19:03.555681 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:03.555658    2560 scope.go:117] "RemoveContainer" containerID="51d5b57fff878dd6b5d34d3dea78ff25bdd429ebecf3a0ebba0952328be5580b"
Sep 17 14:19:03.556005 ip-10-0-129-158 kubenswrapper[2560]: E0917 14:19:03.555984    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51d5b57fff878dd6b5d34d3dea78ff25bdd429ebecf3a0ebba0952328be5580b\": container with ID starting with 51d5b57fff878dd6b5d34d3dea78ff25bdd429ebecf3a0ebba0952328be5580b not found: ID does not exist" containerID="51d5b57fff878dd6b5d34d3dea78ff25bdd429ebecf3a0ebba0952328be5580b"
Sep 17 14:19:03.556086 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:03.556018    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51d5b57fff878dd6b5d34d3dea78ff25bdd429ebecf3a0ebba0952328be5580b"} err="failed to get container status \"51d5b57fff878dd6b5d34d3dea78ff25bdd429ebecf3a0ebba0952328be5580b\": rpc error: code = NotFound desc = could not find container \"51d5b57fff878dd6b5d34d3dea78ff25bdd429ebecf3a0ebba0952328be5580b\": container with ID starting with 51d5b57fff878dd6b5d34d3dea78ff25bdd429ebecf3a0ebba0952328be5580b not found: ID does not exist"
Sep 17 14:19:03.556086 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:03.556044    2560 scope.go:117] "RemoveContainer" containerID="3a6f61ffc5af81c362c8f2df3e6e6509b26ef1fe03f530db443d6b30a487fc7f"
Sep 17 14:19:03.556445 ip-10-0-129-158 kubenswrapper[2560]: E0917 14:19:03.556389    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a6f61ffc5af81c362c8f2df3e6e6509b26ef1fe03f530db443d6b30a487fc7f\": container with ID starting with 3a6f61ffc5af81c362c8f2df3e6e6509b26ef1fe03f530db443d6b30a487fc7f not found: ID does not exist" containerID="3a6f61ffc5af81c362c8f2df3e6e6509b26ef1fe03f530db443d6b30a487fc7f"
Sep 17 14:19:03.556445 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:03.556421    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a6f61ffc5af81c362c8f2df3e6e6509b26ef1fe03f530db443d6b30a487fc7f"} err="failed to get container status \"3a6f61ffc5af81c362c8f2df3e6e6509b26ef1fe03f530db443d6b30a487fc7f\": rpc error: code = NotFound desc = could not find container \"3a6f61ffc5af81c362c8f2df3e6e6509b26ef1fe03f530db443d6b30a487fc7f\": container with ID starting with 3a6f61ffc5af81c362c8f2df3e6e6509b26ef1fe03f530db443d6b30a487fc7f not found: ID does not exist"
Sep 17 14:19:03.604802 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:03.604772    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08fb3245-86c7-4cab-86d9-3b5177ca5101-istio-data\") pod \"08fb3245-86c7-4cab-86d9-3b5177ca5101\" (UID: \"08fb3245-86c7-4cab-86d9-3b5177ca5101\") "
Sep 17 14:19:03.604802 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:03.604805    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08fb3245-86c7-4cab-86d9-3b5177ca5101-istiod-ca-cert\") pod \"08fb3245-86c7-4cab-86d9-3b5177ca5101\" (UID: \"08fb3245-86c7-4cab-86d9-3b5177ca5101\") "
Sep 17 14:19:03.605050 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:03.604846    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08fb3245-86c7-4cab-86d9-3b5177ca5101-istio-podinfo\") pod \"08fb3245-86c7-4cab-86d9-3b5177ca5101\" (UID: \"08fb3245-86c7-4cab-86d9-3b5177ca5101\") "
Sep 17 14:19:03.605050 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:03.604961    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08fb3245-86c7-4cab-86d9-3b5177ca5101-credential-socket\") pod \"08fb3245-86c7-4cab-86d9-3b5177ca5101\" (UID: \"08fb3245-86c7-4cab-86d9-3b5177ca5101\") "
Sep 17 14:19:03.605050 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:03.605007    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08fb3245-86c7-4cab-86d9-3b5177ca5101-workload-certs\") pod \"08fb3245-86c7-4cab-86d9-3b5177ca5101\" (UID: \"08fb3245-86c7-4cab-86d9-3b5177ca5101\") "
Sep 17 14:19:03.605204 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:03.605064    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08fb3245-86c7-4cab-86d9-3b5177ca5101-istio-token\") pod \"08fb3245-86c7-4cab-86d9-3b5177ca5101\" (UID: \"08fb3245-86c7-4cab-86d9-3b5177ca5101\") "
Sep 17 14:19:03.605204 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:03.605086    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08fb3245-86c7-4cab-86d9-3b5177ca5101-istio-envoy\") pod \"08fb3245-86c7-4cab-86d9-3b5177ca5101\" (UID: \"08fb3245-86c7-4cab-86d9-3b5177ca5101\") "
Sep 17 14:19:03.605204 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:03.605083    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08fb3245-86c7-4cab-86d9-3b5177ca5101-istio-data" (OuterVolumeSpecName: "istio-data") pod "08fb3245-86c7-4cab-86d9-3b5177ca5101" (UID: "08fb3245-86c7-4cab-86d9-3b5177ca5101"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:19:03.605204 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:03.605124    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08fb3245-86c7-4cab-86d9-3b5177ca5101-workload-socket\") pod \"08fb3245-86c7-4cab-86d9-3b5177ca5101\" (UID: \"08fb3245-86c7-4cab-86d9-3b5177ca5101\") "
Sep 17 14:19:03.605204 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:03.605182    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mmkxc\" (UniqueName: \"kubernetes.io/projected/08fb3245-86c7-4cab-86d9-3b5177ca5101-kube-api-access-mmkxc\") pod \"08fb3245-86c7-4cab-86d9-3b5177ca5101\" (UID: \"08fb3245-86c7-4cab-86d9-3b5177ca5101\") "
Sep 17 14:19:03.605451 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:03.605226    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08fb3245-86c7-4cab-86d9-3b5177ca5101-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "08fb3245-86c7-4cab-86d9-3b5177ca5101" (UID: "08fb3245-86c7-4cab-86d9-3b5177ca5101"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:19:03.605451 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:03.605332    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08fb3245-86c7-4cab-86d9-3b5177ca5101-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "08fb3245-86c7-4cab-86d9-3b5177ca5101" (UID: "08fb3245-86c7-4cab-86d9-3b5177ca5101"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:19:03.605451 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:03.605351    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08fb3245-86c7-4cab-86d9-3b5177ca5101-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "08fb3245-86c7-4cab-86d9-3b5177ca5101" (UID: "08fb3245-86c7-4cab-86d9-3b5177ca5101"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:19:03.605451 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:03.605455    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08fb3245-86c7-4cab-86d9-3b5177ca5101-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "08fb3245-86c7-4cab-86d9-3b5177ca5101" (UID: "08fb3245-86c7-4cab-86d9-3b5177ca5101"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:19:03.605744 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:03.605726    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08fb3245-86c7-4cab-86d9-3b5177ca5101-istio-data\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:19:03.605807 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:03.605748    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08fb3245-86c7-4cab-86d9-3b5177ca5101-istiod-ca-cert\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:19:03.605807 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:03.605761    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08fb3245-86c7-4cab-86d9-3b5177ca5101-credential-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:19:03.605807 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:03.605775    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08fb3245-86c7-4cab-86d9-3b5177ca5101-workload-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:19:03.605807 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:03.605788    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08fb3245-86c7-4cab-86d9-3b5177ca5101-workload-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:19:03.607171 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:03.607144    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08fb3245-86c7-4cab-86d9-3b5177ca5101-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "08fb3245-86c7-4cab-86d9-3b5177ca5101" (UID: "08fb3245-86c7-4cab-86d9-3b5177ca5101"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:19:03.607273 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:03.607249    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08fb3245-86c7-4cab-86d9-3b5177ca5101-kube-api-access-mmkxc" (OuterVolumeSpecName: "kube-api-access-mmkxc") pod "08fb3245-86c7-4cab-86d9-3b5177ca5101" (UID: "08fb3245-86c7-4cab-86d9-3b5177ca5101"). InnerVolumeSpecName "kube-api-access-mmkxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:19:03.607499 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:03.607482    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/08fb3245-86c7-4cab-86d9-3b5177ca5101-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "08fb3245-86c7-4cab-86d9-3b5177ca5101" (UID: "08fb3245-86c7-4cab-86d9-3b5177ca5101"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 14:19:03.607562 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:03.607509    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08fb3245-86c7-4cab-86d9-3b5177ca5101-istio-token" (OuterVolumeSpecName: "istio-token") pod "08fb3245-86c7-4cab-86d9-3b5177ca5101" (UID: "08fb3245-86c7-4cab-86d9-3b5177ca5101"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:19:03.706982 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:03.706948    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08fb3245-86c7-4cab-86d9-3b5177ca5101-istio-podinfo\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:19:03.706982 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:03.706974    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08fb3245-86c7-4cab-86d9-3b5177ca5101-istio-token\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:19:03.706982 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:03.706984    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08fb3245-86c7-4cab-86d9-3b5177ca5101-istio-envoy\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:19:03.706982 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:03.706992    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmkxc\" (UniqueName: \"kubernetes.io/projected/08fb3245-86c7-4cab-86d9-3b5177ca5101-kube-api-access-mmkxc\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:19:03.859607 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:03.859578    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4"]
Sep 17 14:19:03.863259 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:03.863234    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-96cfc6-openshift-default-557b7b668d-wwvw4"]
Sep 17 14:19:04.138595 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:04.138500    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:04.996790 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:04.996742    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:05.570231 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:05.570196    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08fb3245-86c7-4cab-86d9-3b5177ca5101" path="/var/lib/kubelet/pods/08fb3245-86c7-4cab-86d9-3b5177ca5101/volumes"
Sep 17 14:19:05.905201 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:05.905101    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:06.267503 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:06.267399    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:06.833257 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:06.833219    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:07.777493 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:07.777441    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:08.574119 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:08.574086    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:08.911700 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:08.911613    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:09.128713 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:09.128679    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:09.271547 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:09.271427    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:09.372241 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:09.372198    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:10.368520 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:10.368484    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:11.172600 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:11.172565    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:11.822950 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:11.822919    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:11.852028 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:11.851995    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fe6ee5-6845887b87-fc7sw"]
Sep 17 14:19:11.852214 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:11.852192    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-fe6ee5-6845887b87-fc7sw" podUID="4697b97c-0029-421f-8016-c735369e3da4" containerName="payload-processing" containerID="cri-o://ddb9e3d63b05dc857589355363f7a4748a4e901abcd755e3492c2569228a0fb8" gracePeriod=30
Sep 17 14:19:11.908262 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:11.908143    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:12.010634 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:12.010590    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:12.085469 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:12.085312    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:12.090315 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:12.090259    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fe6ee5-t9qj7"]
Sep 17 14:19:12.092529 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:12.092503    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fe6ee5-t9qj7"]
Sep 17 14:19:12.148188 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:12.148156    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-fe6ee5-6845887b87-fc7sw"
Sep 17 14:19:12.213545 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:12.213503    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4697b97c-0029-421f-8016-c735369e3da4-plugins-config-volume\") pod \"4697b97c-0029-421f-8016-c735369e3da4\" (UID: \"4697b97c-0029-421f-8016-c735369e3da4\") "
Sep 17 14:19:12.213722 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:12.213656    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-52gtt\" (UniqueName: \"kubernetes.io/projected/4697b97c-0029-421f-8016-c735369e3da4-kube-api-access-52gtt\") pod \"4697b97c-0029-421f-8016-c735369e3da4\" (UID: \"4697b97c-0029-421f-8016-c735369e3da4\") "
Sep 17 14:19:12.214307 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:12.214277    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4697b97c-0029-421f-8016-c735369e3da4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4697b97c-0029-421f-8016-c735369e3da4" (UID: "4697b97c-0029-421f-8016-c735369e3da4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:19:12.220215 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:12.220033    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4697b97c-0029-421f-8016-c735369e3da4-kube-api-access-52gtt" (OuterVolumeSpecName: "kube-api-access-52gtt") pod "4697b97c-0029-421f-8016-c735369e3da4" (UID: "4697b97c-0029-421f-8016-c735369e3da4"). InnerVolumeSpecName "kube-api-access-52gtt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:19:12.315799 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:12.315758    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52gtt\" (UniqueName: \"kubernetes.io/projected/4697b97c-0029-421f-8016-c735369e3da4-kube-api-access-52gtt\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:19:12.315799 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:12.315791    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4697b97c-0029-421f-8016-c735369e3da4-plugins-config-volume\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:19:12.514211 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:12.514108    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:12.590153 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:12.590110    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-fe6ee5/test-model-fe6ee5-kserve-7774b7d84f-rn4bs" podUID="fcf756dc-ea28-4e48-b015-704955375715" containerName="main" containerID="cri-o://cb5898f5113991cd53a9e9090624ebda8ba418f501b2a8127b039d2a693f48bb" gracePeriod=45
Sep 17 14:19:12.613069 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:12.613043    2560 generic.go:356] "Generic (PLEG): container finished" podID="4697b97c-0029-421f-8016-c735369e3da4" containerID="ddb9e3d63b05dc857589355363f7a4748a4e901abcd755e3492c2569228a0fb8" exitCode=0
Sep 17 14:19:12.613197 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:12.613083    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-fe6ee5-6845887b87-fc7sw"
Sep 17 14:19:12.613197 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:12.613128    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-fe6ee5-6845887b87-fc7sw" event={"ID":"4697b97c-0029-421f-8016-c735369e3da4","Type":"ContainerDied","Data":"ddb9e3d63b05dc857589355363f7a4748a4e901abcd755e3492c2569228a0fb8"}
Sep 17 14:19:12.613197 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:12.613176    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-fe6ee5-6845887b87-fc7sw" event={"ID":"4697b97c-0029-421f-8016-c735369e3da4","Type":"ContainerDied","Data":"ac9e3e71c3b63fe97fe007832a22a26def442fca2c5d987a673d647bdf611d32"}
Sep 17 14:19:12.613357 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:12.613199    2560 scope.go:117] "RemoveContainer" containerID="ddb9e3d63b05dc857589355363f7a4748a4e901abcd755e3492c2569228a0fb8"
Sep 17 14:19:12.615337 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:12.615310    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:12.666794 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:12.666757    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fe6ee5-6845887b87-fc7sw"]
Sep 17 14:19:12.682152 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:12.682114    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fe6ee5-6845887b87-fc7sw"]
Sep 17 14:19:12.702888 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:12.702860    2560 scope.go:117] "RemoveContainer" containerID="ddb9e3d63b05dc857589355363f7a4748a4e901abcd755e3492c2569228a0fb8"
Sep 17 14:19:12.703265 ip-10-0-129-158 kubenswrapper[2560]: E0917 14:19:12.703244    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddb9e3d63b05dc857589355363f7a4748a4e901abcd755e3492c2569228a0fb8\": container with ID starting with ddb9e3d63b05dc857589355363f7a4748a4e901abcd755e3492c2569228a0fb8 not found: ID does not exist" containerID="ddb9e3d63b05dc857589355363f7a4748a4e901abcd755e3492c2569228a0fb8"
Sep 17 14:19:12.703349 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:12.703279    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddb9e3d63b05dc857589355363f7a4748a4e901abcd755e3492c2569228a0fb8"} err="failed to get container status \"ddb9e3d63b05dc857589355363f7a4748a4e901abcd755e3492c2569228a0fb8\": rpc error: code = NotFound desc = could not find container \"ddb9e3d63b05dc857589355363f7a4748a4e901abcd755e3492c2569228a0fb8\": container with ID starting with ddb9e3d63b05dc857589355363f7a4748a4e901abcd755e3492c2569228a0fb8 not found: ID does not exist"
Sep 17 14:19:12.850224 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:12.850201    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fe6ee5/test-model-fe6ee5-kserve-7774b7d84f-rn4bs"
Sep 17 14:19:12.922733 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:12.922705    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcf756dc-ea28-4e48-b015-704955375715-model-cache\") pod \"fcf756dc-ea28-4e48-b015-704955375715\" (UID: \"fcf756dc-ea28-4e48-b015-704955375715\") "
Sep 17 14:19:12.922892 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:12.922797    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6zzf5\" (UniqueName: \"kubernetes.io/projected/fcf756dc-ea28-4e48-b015-704955375715-kube-api-access-6zzf5\") pod \"fcf756dc-ea28-4e48-b015-704955375715\" (UID: \"fcf756dc-ea28-4e48-b015-704955375715\") "
Sep 17 14:19:12.922892 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:12.922823    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcf756dc-ea28-4e48-b015-704955375715-tls-certs\") pod \"fcf756dc-ea28-4e48-b015-704955375715\" (UID: \"fcf756dc-ea28-4e48-b015-704955375715\") "
Sep 17 14:19:12.922892 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:12.922868    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcf756dc-ea28-4e48-b015-704955375715-home\") pod \"fcf756dc-ea28-4e48-b015-704955375715\" (UID: \"fcf756dc-ea28-4e48-b015-704955375715\") "
Sep 17 14:19:12.923041 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:12.922915    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcf756dc-ea28-4e48-b015-704955375715-dshm\") pod \"fcf756dc-ea28-4e48-b015-704955375715\" (UID: \"fcf756dc-ea28-4e48-b015-704955375715\") "
Sep 17 14:19:12.923041 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:12.922989    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcf756dc-ea28-4e48-b015-704955375715-model-cache" (OuterVolumeSpecName: "model-cache") pod "fcf756dc-ea28-4e48-b015-704955375715" (UID: "fcf756dc-ea28-4e48-b015-704955375715"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:19:12.923041 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:12.923002    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcf756dc-ea28-4e48-b015-704955375715-tmp-dir\") pod \"fcf756dc-ea28-4e48-b015-704955375715\" (UID: \"fcf756dc-ea28-4e48-b015-704955375715\") "
Sep 17 14:19:12.923216 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:12.923187    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcf756dc-ea28-4e48-b015-704955375715-home" (OuterVolumeSpecName: "home") pod "fcf756dc-ea28-4e48-b015-704955375715" (UID: "fcf756dc-ea28-4e48-b015-704955375715"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:19:12.923380 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:12.923273    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcf756dc-ea28-4e48-b015-704955375715-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fcf756dc-ea28-4e48-b015-704955375715" (UID: "fcf756dc-ea28-4e48-b015-704955375715"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:19:12.923561 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:12.923537    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcf756dc-ea28-4e48-b015-704955375715-home\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:19:12.923643 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:12.923562    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcf756dc-ea28-4e48-b015-704955375715-tmp-dir\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:19:12.923643 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:12.923575    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcf756dc-ea28-4e48-b015-704955375715-model-cache\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:19:12.925688 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:12.925660    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcf756dc-ea28-4e48-b015-704955375715-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fcf756dc-ea28-4e48-b015-704955375715" (UID: "fcf756dc-ea28-4e48-b015-704955375715"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:19:12.925785 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:12.925736    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcf756dc-ea28-4e48-b015-704955375715-dshm" (OuterVolumeSpecName: "dshm") pod "fcf756dc-ea28-4e48-b015-704955375715" (UID: "fcf756dc-ea28-4e48-b015-704955375715"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:19:12.925840 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:12.925786    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcf756dc-ea28-4e48-b015-704955375715-kube-api-access-6zzf5" (OuterVolumeSpecName: "kube-api-access-6zzf5") pod "fcf756dc-ea28-4e48-b015-704955375715" (UID: "fcf756dc-ea28-4e48-b015-704955375715"). InnerVolumeSpecName "kube-api-access-6zzf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:19:13.024657 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:13.024567    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zzf5\" (UniqueName: \"kubernetes.io/projected/fcf756dc-ea28-4e48-b015-704955375715-kube-api-access-6zzf5\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:19:13.024657 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:13.024599    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcf756dc-ea28-4e48-b015-704955375715-tls-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:19:13.024657 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:13.024614    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcf756dc-ea28-4e48-b015-704955375715-dshm\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:19:13.571002 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:13.570962    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0aca08da-5a7b-4eda-9a66-10780ccc2da9" path="/var/lib/kubelet/pods/0aca08da-5a7b-4eda-9a66-10780ccc2da9/volumes"
Sep 17 14:19:13.571367 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:13.571352    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4697b97c-0029-421f-8016-c735369e3da4" path="/var/lib/kubelet/pods/4697b97c-0029-421f-8016-c735369e3da4/volumes"
Sep 17 14:19:13.624103 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:13.624063    2560 generic.go:356] "Generic (PLEG): container finished" podID="fcf756dc-ea28-4e48-b015-704955375715" containerID="cb5898f5113991cd53a9e9090624ebda8ba418f501b2a8127b039d2a693f48bb" exitCode=0
Sep 17 14:19:13.624309 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:13.624110    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fe6ee5/test-model-fe6ee5-kserve-7774b7d84f-rn4bs"
Sep 17 14:19:13.624309 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:13.624143    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fe6ee5/test-model-fe6ee5-kserve-7774b7d84f-rn4bs" event={"ID":"fcf756dc-ea28-4e48-b015-704955375715","Type":"ContainerDied","Data":"cb5898f5113991cd53a9e9090624ebda8ba418f501b2a8127b039d2a693f48bb"}
Sep 17 14:19:13.624309 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:13.624195    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fe6ee5/test-model-fe6ee5-kserve-7774b7d84f-rn4bs" event={"ID":"fcf756dc-ea28-4e48-b015-704955375715","Type":"ContainerDied","Data":"6ad56ba5df3d5c955cdf4f3f367102368b500910dc251a6f454e0c83a62a8113"}
Sep 17 14:19:13.624309 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:13.624219    2560 scope.go:117] "RemoveContainer" containerID="cb5898f5113991cd53a9e9090624ebda8ba418f501b2a8127b039d2a693f48bb"
Sep 17 14:19:13.634435 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:13.634414    2560 scope.go:117] "RemoveContainer" containerID="cb5898f5113991cd53a9e9090624ebda8ba418f501b2a8127b039d2a693f48bb"
Sep 17 14:19:13.634834 ip-10-0-129-158 kubenswrapper[2560]: E0917 14:19:13.634813    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb5898f5113991cd53a9e9090624ebda8ba418f501b2a8127b039d2a693f48bb\": container with ID starting with cb5898f5113991cd53a9e9090624ebda8ba418f501b2a8127b039d2a693f48bb not found: ID does not exist" containerID="cb5898f5113991cd53a9e9090624ebda8ba418f501b2a8127b039d2a693f48bb"
Sep 17 14:19:13.634922 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:13.634842    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb5898f5113991cd53a9e9090624ebda8ba418f501b2a8127b039d2a693f48bb"} err="failed to get container status \"cb5898f5113991cd53a9e9090624ebda8ba418f501b2a8127b039d2a693f48bb\": rpc error: code = NotFound desc = could not find container \"cb5898f5113991cd53a9e9090624ebda8ba418f501b2a8127b039d2a693f48bb\": container with ID starting with cb5898f5113991cd53a9e9090624ebda8ba418f501b2a8127b039d2a693f48bb not found: ID does not exist"
Sep 17 14:19:13.643188 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:13.643155    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-fe6ee5/test-model-fe6ee5-kserve-7774b7d84f-rn4bs"]
Sep 17 14:19:13.646341 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:13.646314    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-fe6ee5/test-model-fe6ee5-kserve-7774b7d84f-rn4bs"]
Sep 17 14:19:13.823347 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:13.823254    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:14.244719 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:14.244632    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-ebc24f/noten-model-ebc24f-kserve-69d64dd88c-cxqkj"]
Sep 17 14:19:14.346651 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:14.346609    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:14.447959 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:14.447900    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:15.374367 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:15.374333    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:15.578389 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:15.578350    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcf756dc-ea28-4e48-b015-704955375715" path="/var/lib/kubelet/pods/fcf756dc-ea28-4e48-b015-704955375715/volumes"
Sep 17 14:19:16.222786 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:16.222703    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:16.648713 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:16.648680    2560 generic.go:356] "Generic (PLEG): container finished" podID="98a693c5-386f-4525-b3e1-7f891ef7d68a" containerID="6cb7b2daac5fd38d7fa3dda92140b61a7e71ce7da14c7accd5cab2dd3bd4b32b" exitCode=0
Sep 17 14:19:16.649115 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:16.648752    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h" event={"ID":"98a693c5-386f-4525-b3e1-7f891ef7d68a","Type":"ContainerDied","Data":"6cb7b2daac5fd38d7fa3dda92140b61a7e71ce7da14c7accd5cab2dd3bd4b32b"}
Sep 17 14:19:16.649186 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:16.649173    2560 scope.go:117] "RemoveContainer" containerID="6cb7b2daac5fd38d7fa3dda92140b61a7e71ce7da14c7accd5cab2dd3bd4b32b"
Sep 17 14:19:16.650320 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:16.650293    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:19:16.650388 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:16.650362    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:19:16.650429 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:16.650395    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:19:17.246512 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:17.246451    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:17.658805 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:17.658761    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h" event={"ID":"98a693c5-386f-4525-b3e1-7f891ef7d68a","Type":"ContainerStarted","Data":"ac64c1cf1c6dfcae08f3df2a809ef845eb6d9c8f176d92d7f1add7f46cdcbbeb"}
Sep 17 14:19:18.303271 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:18.303231    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h"
Sep 17 14:19:18.303492 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:18.303455    2560 patch_prober.go:28] interesting pod/e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Sep 17 14:19:18.303550 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:18.303521    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h" podUID="98a693c5-386f-4525-b3e1-7f891ef7d68a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Sep 17 14:19:19.293173 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:19.293131    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:19.303266 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:19.303231    2560 patch_prober.go:28] interesting pod/e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Sep 17 14:19:19.303412 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:19.303303    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h" podUID="98a693c5-386f-4525-b3e1-7f891ef7d68a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Sep 17 14:19:19.378487 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:19.378013    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:19.581291 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:19.580996    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:20.122407 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:20.122376    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:20.302822 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:20.302779    2560 patch_prober.go:28] interesting pod/e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Sep 17 14:19:20.303303 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:20.302845    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h" podUID="98a693c5-386f-4525-b3e1-7f891ef7d68a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Sep 17 14:19:20.474989 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:20.474896    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:20.566028 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:20.565992    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:21.303346 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:21.303312    2560 patch_prober.go:28] interesting pod/e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Sep 17 14:19:21.303761 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:21.303365    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h" podUID="98a693c5-386f-4525-b3e1-7f891ef7d68a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Sep 17 14:19:22.303393 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:22.303355    2560 patch_prober.go:28] interesting pod/e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Sep 17 14:19:22.303814 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:22.303411    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h" podUID="98a693c5-386f-4525-b3e1-7f891ef7d68a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Sep 17 14:19:23.307724 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:23.307691    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h"
Sep 17 14:19:23.308085 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:23.307959    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h"
Sep 17 14:19:23.308716 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:23.308700    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h"
Sep 17 14:19:23.383405 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:23.383371    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:23.948639 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:23.948605    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:24.090691 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:24.090597    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:24.381929 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:24.381891    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:25.579470 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:25.579424    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:26.378510 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:26.378456    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:27.217377 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:27.217342    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:28.392541 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:28.392510    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:28.954489 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:28.954393    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:29.056251 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:29.056203    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:29.151589 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:29.151549    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:29.265113 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:29.265028    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-ebc24f/noten-model-ebc24f-kserve-69d64dd88c-cxqkj" podUID="54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e" containerName="main" containerID="cri-o://2dfed35f65ab3bd51630b3269ddfb03dcc53e1438a434605a68adc2adb94bb08" gracePeriod=45
Sep 17 14:19:29.477569 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:29.477513    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:29.519369 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:29.519308    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-ebc24f/noten-model-ebc24f-kserve-69d64dd88c-cxqkj"
Sep 17 14:19:29.544446 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:29.544417    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:29.621040 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:29.621012    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e-dshm\") pod \"54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e\" (UID: \"54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e\") "
Sep 17 14:19:29.621190 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:29.621049    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8zwrd\" (UniqueName: \"kubernetes.io/projected/54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e-kube-api-access-8zwrd\") pod \"54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e\" (UID: \"54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e\") "
Sep 17 14:19:29.621190 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:29.621076    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e-tmp-dir\") pod \"54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e\" (UID: \"54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e\") "
Sep 17 14:19:29.621190 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:29.621180    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e-model-cache\") pod \"54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e\" (UID: \"54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e\") "
Sep 17 14:19:29.621373 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:29.621229    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e-tls-certs\") pod \"54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e\" (UID: \"54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e\") "
Sep 17 14:19:29.621373 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:29.621340    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e-home\") pod \"54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e\" (UID: \"54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e\") "
Sep 17 14:19:29.621547 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:29.621515    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e" (UID: "54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:19:29.621615 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:29.621534    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e-model-cache" (OuterVolumeSpecName: "model-cache") pod "54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e" (UID: "54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:19:29.621696 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:29.621674    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e-home" (OuterVolumeSpecName: "home") pod "54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e" (UID: "54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:19:29.621824 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:29.621803    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e-home\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:19:29.621896 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:29.621855    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e-tmp-dir\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:19:29.621896 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:29.621873    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e-model-cache\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:19:29.623279 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:29.623259    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e-dshm" (OuterVolumeSpecName: "dshm") pod "54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e" (UID: "54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:19:29.623391 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:29.623297    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e-kube-api-access-8zwrd" (OuterVolumeSpecName: "kube-api-access-8zwrd") pod "54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e" (UID: "54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e"). InnerVolumeSpecName "kube-api-access-8zwrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:19:29.623391 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:29.623356    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e" (UID: "54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:19:29.723205 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:29.723151    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e-tls-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:19:29.723205 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:29.723189    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e-dshm\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:19:29.723205 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:29.723202    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zwrd\" (UniqueName: \"kubernetes.io/projected/54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e-kube-api-access-8zwrd\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:19:29.758115 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:29.758085    2560 generic.go:356] "Generic (PLEG): container finished" podID="54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e" containerID="2dfed35f65ab3bd51630b3269ddfb03dcc53e1438a434605a68adc2adb94bb08" exitCode=0
Sep 17 14:19:29.758283 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:29.758139    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-ebc24f/noten-model-ebc24f-kserve-69d64dd88c-cxqkj"
Sep 17 14:19:29.758283 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:29.758153    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-ebc24f/noten-model-ebc24f-kserve-69d64dd88c-cxqkj" event={"ID":"54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e","Type":"ContainerDied","Data":"2dfed35f65ab3bd51630b3269ddfb03dcc53e1438a434605a68adc2adb94bb08"}
Sep 17 14:19:29.758283 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:29.758184    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-ebc24f/noten-model-ebc24f-kserve-69d64dd88c-cxqkj" event={"ID":"54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e","Type":"ContainerDied","Data":"647c4fc778e71ad3337312132268e348165d7a71c9a02d7392cd0eb3fdaea4ad"}
Sep 17 14:19:29.758283 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:29.758200    2560 scope.go:117] "RemoveContainer" containerID="2dfed35f65ab3bd51630b3269ddfb03dcc53e1438a434605a68adc2adb94bb08"
Sep 17 14:19:29.768770 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:29.768750    2560 scope.go:117] "RemoveContainer" containerID="2dfed35f65ab3bd51630b3269ddfb03dcc53e1438a434605a68adc2adb94bb08"
Sep 17 14:19:29.769023 ip-10-0-129-158 kubenswrapper[2560]: E0917 14:19:29.769004    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dfed35f65ab3bd51630b3269ddfb03dcc53e1438a434605a68adc2adb94bb08\": container with ID starting with 2dfed35f65ab3bd51630b3269ddfb03dcc53e1438a434605a68adc2adb94bb08 not found: ID does not exist" containerID="2dfed35f65ab3bd51630b3269ddfb03dcc53e1438a434605a68adc2adb94bb08"
Sep 17 14:19:29.769076 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:29.769029    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dfed35f65ab3bd51630b3269ddfb03dcc53e1438a434605a68adc2adb94bb08"} err="failed to get container status \"2dfed35f65ab3bd51630b3269ddfb03dcc53e1438a434605a68adc2adb94bb08\": rpc error: code = NotFound desc = could not find container \"2dfed35f65ab3bd51630b3269ddfb03dcc53e1438a434605a68adc2adb94bb08\": container with ID starting with 2dfed35f65ab3bd51630b3269ddfb03dcc53e1438a434605a68adc2adb94bb08 not found: ID does not exist"
Sep 17 14:19:29.782021 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:29.781970    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-ebc24f/noten-model-ebc24f-kserve-69d64dd88c-cxqkj"]
Sep 17 14:19:29.788233 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:29.788212    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-ebc24f/noten-model-ebc24f-kserve-69d64dd88c-cxqkj"]
Sep 17 14:19:31.570168 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:31.570128    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e" path="/var/lib/kubelet/pods/54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e/volumes"
Sep 17 14:19:31.741649 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:31.741613    2560 scope.go:117] "RemoveContainer" containerID="383110f642adc0f506333b66d5c00e2faada7b0f2e36848ee38cd3004cad8bcf"
Sep 17 14:19:31.751158 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:31.751133    2560 scope.go:117] "RemoveContainer" containerID="c881b46a28f4fa8ca75a4a48a1a008737841f80c66328bf9ae03c4723cda492a"
Sep 17 14:19:38.702695 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:38.702664    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-d9b17d/distinct-2-w0-d9b17d-kserve-56497687d9-7x45h"]
Sep 17 14:19:38.703142 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:38.703129    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e" containerName="main"
Sep 17 14:19:38.703142 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:38.703143    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e" containerName="main"
Sep 17 14:19:38.703218 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:38.703160    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcf756dc-ea28-4e48-b015-704955375715" containerName="main"
Sep 17 14:19:38.703218 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:38.703165    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcf756dc-ea28-4e48-b015-704955375715" containerName="main"
Sep 17 14:19:38.703218 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:38.703177    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08fb3245-86c7-4cab-86d9-3b5177ca5101" containerName="istio-proxy"
Sep 17 14:19:38.703218 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:38.703182    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="08fb3245-86c7-4cab-86d9-3b5177ca5101" containerName="istio-proxy"
Sep 17 14:19:38.703218 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:38.703195    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08fb3245-86c7-4cab-86d9-3b5177ca5101" containerName="istio-proxy"
Sep 17 14:19:38.703218 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:38.703200    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="08fb3245-86c7-4cab-86d9-3b5177ca5101" containerName="istio-proxy"
Sep 17 14:19:38.703218 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:38.703205    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4697b97c-0029-421f-8016-c735369e3da4" containerName="payload-processing"
Sep 17 14:19:38.703218 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:38.703211    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4697b97c-0029-421f-8016-c735369e3da4" containerName="payload-processing"
Sep 17 14:19:38.703218 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:38.703216    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0aca08da-5a7b-4eda-9a66-10780ccc2da9" containerName="revoke-keys"
Sep 17 14:19:38.703218 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:38.703221    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0aca08da-5a7b-4eda-9a66-10780ccc2da9" containerName="revoke-keys"
Sep 17 14:19:38.703546 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:38.703286    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="54d1d86b-f3ee-4ce2-9e9c-135c8ea53e1e" containerName="main"
Sep 17 14:19:38.703546 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:38.703297    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="08fb3245-86c7-4cab-86d9-3b5177ca5101" containerName="istio-proxy"
Sep 17 14:19:38.703546 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:38.703304    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcf756dc-ea28-4e48-b015-704955375715" containerName="main"
Sep 17 14:19:38.703546 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:38.703311    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4697b97c-0029-421f-8016-c735369e3da4" containerName="payload-processing"
Sep 17 14:19:38.703546 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:38.703318    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="08fb3245-86c7-4cab-86d9-3b5177ca5101" containerName="istio-proxy"
Sep 17 14:19:38.703546 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:38.703323    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0aca08da-5a7b-4eda-9a66-10780ccc2da9" containerName="revoke-keys"
Sep 17 14:19:38.708060 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:38.708038    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d9b17d/distinct-2-w0-d9b17d-kserve-56497687d9-7x45h"
Sep 17 14:19:38.711085 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:38.711061    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d9b17d\"/\"distinct-2-w0-d9b17d-kserve-self-signed-certs\""
Sep 17 14:19:38.715539 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:38.715514    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d9b17d/distinct-2-w0-d9b17d-kserve-56497687d9-7x45h"]
Sep 17 14:19:38.732426 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:38.732387    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:38.820411 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:38.820380    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6353689b-97e8-457d-b0e1-8f167deff124-tls-certs\") pod \"distinct-2-w0-d9b17d-kserve-56497687d9-7x45h\" (UID: \"6353689b-97e8-457d-b0e1-8f167deff124\") " pod="e2e-models-e2e-worker-w0-d9b17d/distinct-2-w0-d9b17d-kserve-56497687d9-7x45h"
Sep 17 14:19:38.820624 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:38.820450    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6353689b-97e8-457d-b0e1-8f167deff124-home\") pod \"distinct-2-w0-d9b17d-kserve-56497687d9-7x45h\" (UID: \"6353689b-97e8-457d-b0e1-8f167deff124\") " pod="e2e-models-e2e-worker-w0-d9b17d/distinct-2-w0-d9b17d-kserve-56497687d9-7x45h"
Sep 17 14:19:38.820624 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:38.820603    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6353689b-97e8-457d-b0e1-8f167deff124-model-cache\") pod \"distinct-2-w0-d9b17d-kserve-56497687d9-7x45h\" (UID: \"6353689b-97e8-457d-b0e1-8f167deff124\") " pod="e2e-models-e2e-worker-w0-d9b17d/distinct-2-w0-d9b17d-kserve-56497687d9-7x45h"
Sep 17 14:19:38.820760 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:38.820697    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6353689b-97e8-457d-b0e1-8f167deff124-dshm\") pod \"distinct-2-w0-d9b17d-kserve-56497687d9-7x45h\" (UID: \"6353689b-97e8-457d-b0e1-8f167deff124\") " pod="e2e-models-e2e-worker-w0-d9b17d/distinct-2-w0-d9b17d-kserve-56497687d9-7x45h"
Sep 17 14:19:38.820760 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:38.820747    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6353689b-97e8-457d-b0e1-8f167deff124-tmp-dir\") pod \"distinct-2-w0-d9b17d-kserve-56497687d9-7x45h\" (UID: \"6353689b-97e8-457d-b0e1-8f167deff124\") " pod="e2e-models-e2e-worker-w0-d9b17d/distinct-2-w0-d9b17d-kserve-56497687d9-7x45h"
Sep 17 14:19:38.820855 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:38.820827    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjxj5\" (UniqueName: \"kubernetes.io/projected/6353689b-97e8-457d-b0e1-8f167deff124-kube-api-access-tjxj5\") pod \"distinct-2-w0-d9b17d-kserve-56497687d9-7x45h\" (UID: \"6353689b-97e8-457d-b0e1-8f167deff124\") " pod="e2e-models-e2e-worker-w0-d9b17d/distinct-2-w0-d9b17d-kserve-56497687d9-7x45h"
Sep 17 14:19:38.922038 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:38.922007    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6353689b-97e8-457d-b0e1-8f167deff124-home\") pod \"distinct-2-w0-d9b17d-kserve-56497687d9-7x45h\" (UID: \"6353689b-97e8-457d-b0e1-8f167deff124\") " pod="e2e-models-e2e-worker-w0-d9b17d/distinct-2-w0-d9b17d-kserve-56497687d9-7x45h"
Sep 17 14:19:38.922183 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:38.922070    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6353689b-97e8-457d-b0e1-8f167deff124-model-cache\") pod \"distinct-2-w0-d9b17d-kserve-56497687d9-7x45h\" (UID: \"6353689b-97e8-457d-b0e1-8f167deff124\") " pod="e2e-models-e2e-worker-w0-d9b17d/distinct-2-w0-d9b17d-kserve-56497687d9-7x45h"
Sep 17 14:19:38.922183 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:38.922139    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6353689b-97e8-457d-b0e1-8f167deff124-dshm\") pod \"distinct-2-w0-d9b17d-kserve-56497687d9-7x45h\" (UID: \"6353689b-97e8-457d-b0e1-8f167deff124\") " pod="e2e-models-e2e-worker-w0-d9b17d/distinct-2-w0-d9b17d-kserve-56497687d9-7x45h"
Sep 17 14:19:38.922291 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:38.922182    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6353689b-97e8-457d-b0e1-8f167deff124-tmp-dir\") pod \"distinct-2-w0-d9b17d-kserve-56497687d9-7x45h\" (UID: \"6353689b-97e8-457d-b0e1-8f167deff124\") " pod="e2e-models-e2e-worker-w0-d9b17d/distinct-2-w0-d9b17d-kserve-56497687d9-7x45h"
Sep 17 14:19:38.922291 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:38.922228    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjxj5\" (UniqueName: \"kubernetes.io/projected/6353689b-97e8-457d-b0e1-8f167deff124-kube-api-access-tjxj5\") pod \"distinct-2-w0-d9b17d-kserve-56497687d9-7x45h\" (UID: \"6353689b-97e8-457d-b0e1-8f167deff124\") " pod="e2e-models-e2e-worker-w0-d9b17d/distinct-2-w0-d9b17d-kserve-56497687d9-7x45h"
Sep 17 14:19:38.922291 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:38.922269    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6353689b-97e8-457d-b0e1-8f167deff124-tls-certs\") pod \"distinct-2-w0-d9b17d-kserve-56497687d9-7x45h\" (UID: \"6353689b-97e8-457d-b0e1-8f167deff124\") " pod="e2e-models-e2e-worker-w0-d9b17d/distinct-2-w0-d9b17d-kserve-56497687d9-7x45h"
Sep 17 14:19:38.922437 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:38.922393    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6353689b-97e8-457d-b0e1-8f167deff124-home\") pod \"distinct-2-w0-d9b17d-kserve-56497687d9-7x45h\" (UID: \"6353689b-97e8-457d-b0e1-8f167deff124\") " pod="e2e-models-e2e-worker-w0-d9b17d/distinct-2-w0-d9b17d-kserve-56497687d9-7x45h"
Sep 17 14:19:38.922526 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:38.922453    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6353689b-97e8-457d-b0e1-8f167deff124-model-cache\") pod \"distinct-2-w0-d9b17d-kserve-56497687d9-7x45h\" (UID: \"6353689b-97e8-457d-b0e1-8f167deff124\") " pod="e2e-models-e2e-worker-w0-d9b17d/distinct-2-w0-d9b17d-kserve-56497687d9-7x45h"
Sep 17 14:19:38.922646 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:38.922626    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6353689b-97e8-457d-b0e1-8f167deff124-tmp-dir\") pod \"distinct-2-w0-d9b17d-kserve-56497687d9-7x45h\" (UID: \"6353689b-97e8-457d-b0e1-8f167deff124\") " pod="e2e-models-e2e-worker-w0-d9b17d/distinct-2-w0-d9b17d-kserve-56497687d9-7x45h"
Sep 17 14:19:38.924349 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:38.924329    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6353689b-97e8-457d-b0e1-8f167deff124-dshm\") pod \"distinct-2-w0-d9b17d-kserve-56497687d9-7x45h\" (UID: \"6353689b-97e8-457d-b0e1-8f167deff124\") " pod="e2e-models-e2e-worker-w0-d9b17d/distinct-2-w0-d9b17d-kserve-56497687d9-7x45h"
Sep 17 14:19:38.924727 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:38.924708    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6353689b-97e8-457d-b0e1-8f167deff124-tls-certs\") pod \"distinct-2-w0-d9b17d-kserve-56497687d9-7x45h\" (UID: \"6353689b-97e8-457d-b0e1-8f167deff124\") " pod="e2e-models-e2e-worker-w0-d9b17d/distinct-2-w0-d9b17d-kserve-56497687d9-7x45h"
Sep 17 14:19:38.932121 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:38.932091    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjxj5\" (UniqueName: \"kubernetes.io/projected/6353689b-97e8-457d-b0e1-8f167deff124-kube-api-access-tjxj5\") pod \"distinct-2-w0-d9b17d-kserve-56497687d9-7x45h\" (UID: \"6353689b-97e8-457d-b0e1-8f167deff124\") " pod="e2e-models-e2e-worker-w0-d9b17d/distinct-2-w0-d9b17d-kserve-56497687d9-7x45h"
Sep 17 14:19:39.019744 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:39.019654    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d9b17d/distinct-2-w0-d9b17d-kserve-56497687d9-7x45h"
Sep 17 14:19:39.295698 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:39.295673    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d9b17d/distinct-2-w0-d9b17d-kserve-56497687d9-7x45h"]
Sep 17 14:19:39.296596 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:19:39.296569    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6353689b_97e8_457d_b0e1_8f167deff124.slice/crio-1d70a38a4e0ce2aab3fcd58b4471a07659424feda8b7cc3a5eac2c8456804118 WatchSource:0}: Error finding container 1d70a38a4e0ce2aab3fcd58b4471a07659424feda8b7cc3a5eac2c8456804118: Status 404 returned error can't find the container with id 1d70a38a4e0ce2aab3fcd58b4471a07659424feda8b7cc3a5eac2c8456804118
Sep 17 14:19:39.833138 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:39.833095    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d9b17d/distinct-2-w0-d9b17d-kserve-56497687d9-7x45h" event={"ID":"6353689b-97e8-457d-b0e1-8f167deff124","Type":"ContainerStarted","Data":"54d1c47ad4ebeab01653e53e688eb0def08c7e7b993eccbbb42d95f572b3ff87"}
Sep 17 14:19:39.833539 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:39.833148    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d9b17d/distinct-2-w0-d9b17d-kserve-56497687d9-7x45h" event={"ID":"6353689b-97e8-457d-b0e1-8f167deff124","Type":"ContainerStarted","Data":"1d70a38a4e0ce2aab3fcd58b4471a07659424feda8b7cc3a5eac2c8456804118"}
Sep 17 14:19:39.854554 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:39.854502    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-d9b17d/distinct-2-w0-d9b17d-kserve-56497687d9-7x45h" podStartSLOduration=1.854456683 podStartE2EDuration="1.854456683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:19:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:19:39.851063518 +0000 UTC m=+1448.793207163" watchObservedRunningTime="2026-09-17 14:19:39.854456683 +0000 UTC m=+1448.796600332"
Sep 17 14:19:41.739113 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:41.739081    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:46.522355 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:46.522280    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-079b38/e2e-conflict-b-9812c892-kserve-96d9c4849-ghjmf"]
Sep 17 14:19:46.526566 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:46.526542    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-079b38/e2e-conflict-b-9812c892-kserve-96d9c4849-ghjmf"
Sep 17 14:19:46.529033 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:46.529011    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-079b38\"/\"e2e-conflict-b-9812c892-kserve-self-signed-certs\""
Sep 17 14:19:46.533833 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:46.533809    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-079b38/e2e-conflict-b-9812c892-kserve-96d9c4849-ghjmf"]
Sep 17 14:19:46.558529 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:46.558416    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:46.600273 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:46.600236    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jslws\" (UniqueName: \"kubernetes.io/projected/3ea78297-9a5f-4d0c-9912-a4fd415043ec-kube-api-access-jslws\") pod \"e2e-conflict-b-9812c892-kserve-96d9c4849-ghjmf\" (UID: \"3ea78297-9a5f-4d0c-9912-a4fd415043ec\") " pod="e2e-models-e2e-worker-w4-079b38/e2e-conflict-b-9812c892-kserve-96d9c4849-ghjmf"
Sep 17 14:19:46.600455 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:46.600298    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ea78297-9a5f-4d0c-9912-a4fd415043ec-tmp-dir\") pod \"e2e-conflict-b-9812c892-kserve-96d9c4849-ghjmf\" (UID: \"3ea78297-9a5f-4d0c-9912-a4fd415043ec\") " pod="e2e-models-e2e-worker-w4-079b38/e2e-conflict-b-9812c892-kserve-96d9c4849-ghjmf"
Sep 17 14:19:46.600455 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:46.600335    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ea78297-9a5f-4d0c-9912-a4fd415043ec-tls-certs\") pod \"e2e-conflict-b-9812c892-kserve-96d9c4849-ghjmf\" (UID: \"3ea78297-9a5f-4d0c-9912-a4fd415043ec\") " pod="e2e-models-e2e-worker-w4-079b38/e2e-conflict-b-9812c892-kserve-96d9c4849-ghjmf"
Sep 17 14:19:46.600455 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:46.600366    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ea78297-9a5f-4d0c-9912-a4fd415043ec-home\") pod \"e2e-conflict-b-9812c892-kserve-96d9c4849-ghjmf\" (UID: \"3ea78297-9a5f-4d0c-9912-a4fd415043ec\") " pod="e2e-models-e2e-worker-w4-079b38/e2e-conflict-b-9812c892-kserve-96d9c4849-ghjmf"
Sep 17 14:19:46.600455 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:46.600456    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ea78297-9a5f-4d0c-9912-a4fd415043ec-model-cache\") pod \"e2e-conflict-b-9812c892-kserve-96d9c4849-ghjmf\" (UID: \"3ea78297-9a5f-4d0c-9912-a4fd415043ec\") " pod="e2e-models-e2e-worker-w4-079b38/e2e-conflict-b-9812c892-kserve-96d9c4849-ghjmf"
Sep 17 14:19:46.600740 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:46.600558    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ea78297-9a5f-4d0c-9912-a4fd415043ec-dshm\") pod \"e2e-conflict-b-9812c892-kserve-96d9c4849-ghjmf\" (UID: \"3ea78297-9a5f-4d0c-9912-a4fd415043ec\") " pod="e2e-models-e2e-worker-w4-079b38/e2e-conflict-b-9812c892-kserve-96d9c4849-ghjmf"
Sep 17 14:19:46.702259 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:46.702208    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jslws\" (UniqueName: \"kubernetes.io/projected/3ea78297-9a5f-4d0c-9912-a4fd415043ec-kube-api-access-jslws\") pod \"e2e-conflict-b-9812c892-kserve-96d9c4849-ghjmf\" (UID: \"3ea78297-9a5f-4d0c-9912-a4fd415043ec\") " pod="e2e-models-e2e-worker-w4-079b38/e2e-conflict-b-9812c892-kserve-96d9c4849-ghjmf"
Sep 17 14:19:46.702557 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:46.702294    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ea78297-9a5f-4d0c-9912-a4fd415043ec-tmp-dir\") pod \"e2e-conflict-b-9812c892-kserve-96d9c4849-ghjmf\" (UID: \"3ea78297-9a5f-4d0c-9912-a4fd415043ec\") " pod="e2e-models-e2e-worker-w4-079b38/e2e-conflict-b-9812c892-kserve-96d9c4849-ghjmf"
Sep 17 14:19:46.702557 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:46.702329    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ea78297-9a5f-4d0c-9912-a4fd415043ec-tls-certs\") pod \"e2e-conflict-b-9812c892-kserve-96d9c4849-ghjmf\" (UID: \"3ea78297-9a5f-4d0c-9912-a4fd415043ec\") " pod="e2e-models-e2e-worker-w4-079b38/e2e-conflict-b-9812c892-kserve-96d9c4849-ghjmf"
Sep 17 14:19:46.702557 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:46.702377    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ea78297-9a5f-4d0c-9912-a4fd415043ec-home\") pod \"e2e-conflict-b-9812c892-kserve-96d9c4849-ghjmf\" (UID: \"3ea78297-9a5f-4d0c-9912-a4fd415043ec\") " pod="e2e-models-e2e-worker-w4-079b38/e2e-conflict-b-9812c892-kserve-96d9c4849-ghjmf"
Sep 17 14:19:46.702557 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:46.702484    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ea78297-9a5f-4d0c-9912-a4fd415043ec-model-cache\") pod \"e2e-conflict-b-9812c892-kserve-96d9c4849-ghjmf\" (UID: \"3ea78297-9a5f-4d0c-9912-a4fd415043ec\") " pod="e2e-models-e2e-worker-w4-079b38/e2e-conflict-b-9812c892-kserve-96d9c4849-ghjmf"
Sep 17 14:19:46.702557 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:46.702552    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ea78297-9a5f-4d0c-9912-a4fd415043ec-dshm\") pod \"e2e-conflict-b-9812c892-kserve-96d9c4849-ghjmf\" (UID: \"3ea78297-9a5f-4d0c-9912-a4fd415043ec\") " pod="e2e-models-e2e-worker-w4-079b38/e2e-conflict-b-9812c892-kserve-96d9c4849-ghjmf"
Sep 17 14:19:46.703357 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:46.703305    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ea78297-9a5f-4d0c-9912-a4fd415043ec-model-cache\") pod \"e2e-conflict-b-9812c892-kserve-96d9c4849-ghjmf\" (UID: \"3ea78297-9a5f-4d0c-9912-a4fd415043ec\") " pod="e2e-models-e2e-worker-w4-079b38/e2e-conflict-b-9812c892-kserve-96d9c4849-ghjmf"
Sep 17 14:19:46.703357 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:46.703334    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ea78297-9a5f-4d0c-9912-a4fd415043ec-home\") pod \"e2e-conflict-b-9812c892-kserve-96d9c4849-ghjmf\" (UID: \"3ea78297-9a5f-4d0c-9912-a4fd415043ec\") " pod="e2e-models-e2e-worker-w4-079b38/e2e-conflict-b-9812c892-kserve-96d9c4849-ghjmf"
Sep 17 14:19:46.704167 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:46.704145    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ea78297-9a5f-4d0c-9912-a4fd415043ec-tmp-dir\") pod \"e2e-conflict-b-9812c892-kserve-96d9c4849-ghjmf\" (UID: \"3ea78297-9a5f-4d0c-9912-a4fd415043ec\") " pod="e2e-models-e2e-worker-w4-079b38/e2e-conflict-b-9812c892-kserve-96d9c4849-ghjmf"
Sep 17 14:19:46.705966 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:46.705898    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ea78297-9a5f-4d0c-9912-a4fd415043ec-dshm\") pod \"e2e-conflict-b-9812c892-kserve-96d9c4849-ghjmf\" (UID: \"3ea78297-9a5f-4d0c-9912-a4fd415043ec\") " pod="e2e-models-e2e-worker-w4-079b38/e2e-conflict-b-9812c892-kserve-96d9c4849-ghjmf"
Sep 17 14:19:46.706180 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:46.706158    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ea78297-9a5f-4d0c-9912-a4fd415043ec-tls-certs\") pod \"e2e-conflict-b-9812c892-kserve-96d9c4849-ghjmf\" (UID: \"3ea78297-9a5f-4d0c-9912-a4fd415043ec\") " pod="e2e-models-e2e-worker-w4-079b38/e2e-conflict-b-9812c892-kserve-96d9c4849-ghjmf"
Sep 17 14:19:46.712118 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:46.712069    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jslws\" (UniqueName: \"kubernetes.io/projected/3ea78297-9a5f-4d0c-9912-a4fd415043ec-kube-api-access-jslws\") pod \"e2e-conflict-b-9812c892-kserve-96d9c4849-ghjmf\" (UID: \"3ea78297-9a5f-4d0c-9912-a4fd415043ec\") " pod="e2e-models-e2e-worker-w4-079b38/e2e-conflict-b-9812c892-kserve-96d9c4849-ghjmf"
Sep 17 14:19:46.837560 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:46.837521    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-079b38/e2e-conflict-b-9812c892-kserve-96d9c4849-ghjmf"
Sep 17 14:19:47.125164 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:47.125138    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-079b38/e2e-conflict-b-9812c892-kserve-96d9c4849-ghjmf"]
Sep 17 14:19:47.127085 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:19:47.127052    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3ea78297_9a5f_4d0c_9912_a4fd415043ec.slice/crio-bb31739b481e0eaa25cd4d246d4ea660c0b17855ffa74d12ec6bac038b5daa57 WatchSource:0}: Error finding container bb31739b481e0eaa25cd4d246d4ea660c0b17855ffa74d12ec6bac038b5daa57: Status 404 returned error can't find the container with id bb31739b481e0eaa25cd4d246d4ea660c0b17855ffa74d12ec6bac038b5daa57
Sep 17 14:19:47.896079 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:47.896043    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-079b38/e2e-conflict-b-9812c892-kserve-96d9c4849-ghjmf" event={"ID":"3ea78297-9a5f-4d0c-9912-a4fd415043ec","Type":"ContainerStarted","Data":"5538cfd0116c172add529ba3fe0797fd7ad84ac5cdc4120a8ff62c65258e6f38"}
Sep 17 14:19:47.896517 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:47.896089    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-079b38/e2e-conflict-b-9812c892-kserve-96d9c4849-ghjmf" event={"ID":"3ea78297-9a5f-4d0c-9912-a4fd415043ec","Type":"ContainerStarted","Data":"bb31739b481e0eaa25cd4d246d4ea660c0b17855ffa74d12ec6bac038b5daa57"}
Sep 17 14:19:47.915696 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:47.915647    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-079b38/e2e-conflict-b-9812c892-kserve-96d9c4849-ghjmf" podStartSLOduration=1.915630702 podStartE2EDuration="1.915630702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:19:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:19:47.914586629 +0000 UTC m=+1456.856730276" watchObservedRunningTime="2026-09-17 14:19:47.915630702 +0000 UTC m=+1456.857774344"
Sep 17 14:19:49.019926 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:49.019879    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d9b17d/distinct-2-w0-d9b17d-kserve-56497687d9-7x45h"
Sep 17 14:19:49.020357 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:49.019960    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-d9b17d/distinct-2-w0-d9b17d-kserve-56497687d9-7x45h"
Sep 17 14:19:49.032519 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:49.032491    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-d9b17d/distinct-2-w0-d9b17d-kserve-56497687d9-7x45h"
Sep 17 14:19:49.920766 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:49.920738    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d9b17d/distinct-2-w0-d9b17d-kserve-56497687d9-7x45h"
Sep 17 14:19:52.176438 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:52.176406    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:52.660405 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:52.660367    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:53.458547 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:53.458520    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:53.942861 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:53.942823    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:54.150377 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:54.150344    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:54.559890 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:54.559850    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk"]
Sep 17 14:19:54.565737 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:54.565708    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk"
Sep 17 14:19:54.569400 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:54.569381    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-b79988-openshift-default-dockercfg-8hh8h\""
Sep 17 14:19:54.596317 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:54.596290    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk"]
Sep 17 14:19:54.687256 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:54.687199    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c-istio-envoy\") pod \"e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk\" (UID: \"00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c\") " pod="openshift-ingress/e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk"
Sep 17 14:19:54.687436 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:54.687261    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c-istio-podinfo\") pod \"e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk\" (UID: \"00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c\") " pod="openshift-ingress/e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk"
Sep 17 14:19:54.687436 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:54.687304    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xkjf\" (UniqueName: \"kubernetes.io/projected/00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c-kube-api-access-5xkjf\") pod \"e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk\" (UID: \"00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c\") " pod="openshift-ingress/e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk"
Sep 17 14:19:54.687436 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:54.687403    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c-credential-socket\") pod \"e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk\" (UID: \"00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c\") " pod="openshift-ingress/e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk"
Sep 17 14:19:54.687771 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:54.687456    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c-workload-certs\") pod \"e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk\" (UID: \"00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c\") " pod="openshift-ingress/e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk"
Sep 17 14:19:54.687771 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:54.687521    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c-istiod-ca-cert\") pod \"e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk\" (UID: \"00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c\") " pod="openshift-ingress/e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk"
Sep 17 14:19:54.687771 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:54.687588    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c-workload-socket\") pod \"e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk\" (UID: \"00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c\") " pod="openshift-ingress/e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk"
Sep 17 14:19:54.687771 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:54.687636    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c-istio-data\") pod \"e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk\" (UID: \"00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c\") " pod="openshift-ingress/e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk"
Sep 17 14:19:54.687771 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:54.687667    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c-istio-token\") pod \"e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk\" (UID: \"00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c\") " pod="openshift-ingress/e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk"
Sep 17 14:19:54.701053 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:54.701011    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:54.789048 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:54.789011    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c-credential-socket\") pod \"e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk\" (UID: \"00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c\") " pod="openshift-ingress/e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk"
Sep 17 14:19:54.789279 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:54.789060    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c-workload-certs\") pod \"e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk\" (UID: \"00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c\") " pod="openshift-ingress/e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk"
Sep 17 14:19:54.789279 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:54.789088    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c-istiod-ca-cert\") pod \"e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk\" (UID: \"00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c\") " pod="openshift-ingress/e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk"
Sep 17 14:19:54.789279 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:54.789138    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c-workload-socket\") pod \"e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk\" (UID: \"00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c\") " pod="openshift-ingress/e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk"
Sep 17 14:19:54.789279 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:54.789176    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c-istio-data\") pod \"e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk\" (UID: \"00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c\") " pod="openshift-ingress/e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk"
Sep 17 14:19:54.789279 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:54.789205    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c-istio-token\") pod \"e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk\" (UID: \"00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c\") " pod="openshift-ingress/e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk"
Sep 17 14:19:54.789279 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:54.789271    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c-istio-envoy\") pod \"e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk\" (UID: \"00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c\") " pod="openshift-ingress/e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk"
Sep 17 14:19:54.789704 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:54.789307    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c-istio-podinfo\") pod \"e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk\" (UID: \"00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c\") " pod="openshift-ingress/e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk"
Sep 17 14:19:54.789704 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:54.789344    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xkjf\" (UniqueName: \"kubernetes.io/projected/00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c-kube-api-access-5xkjf\") pod \"e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk\" (UID: \"00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c\") " pod="openshift-ingress/e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk"
Sep 17 14:19:54.789704 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:54.789609    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c-istio-data\") pod \"e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk\" (UID: \"00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c\") " pod="openshift-ingress/e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk"
Sep 17 14:19:54.790135 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:54.790112    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c-workload-certs\") pod \"e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk\" (UID: \"00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c\") " pod="openshift-ingress/e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk"
Sep 17 14:19:54.790545 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:54.790357    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c-credential-socket\") pod \"e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk\" (UID: \"00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c\") " pod="openshift-ingress/e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk"
Sep 17 14:19:54.790545 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:54.790361    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c-istiod-ca-cert\") pod \"e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk\" (UID: \"00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c\") " pod="openshift-ingress/e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk"
Sep 17 14:19:54.790686 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:54.790586    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c-workload-socket\") pod \"e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk\" (UID: \"00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c\") " pod="openshift-ingress/e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk"
Sep 17 14:19:54.792745 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:54.792719    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c-istio-podinfo\") pod \"e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk\" (UID: \"00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c\") " pod="openshift-ingress/e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk"
Sep 17 14:19:54.792865 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:54.792831    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c-istio-envoy\") pod \"e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk\" (UID: \"00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c\") " pod="openshift-ingress/e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk"
Sep 17 14:19:54.797824 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:54.797799    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c-istio-token\") pod \"e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk\" (UID: \"00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c\") " pod="openshift-ingress/e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk"
Sep 17 14:19:54.802722 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:54.802690    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xkjf\" (UniqueName: \"kubernetes.io/projected/00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c-kube-api-access-5xkjf\") pod \"e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk\" (UID: \"00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c\") " pod="openshift-ingress/e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk"
Sep 17 14:19:54.802838 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:54.802818    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:54.877206 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:54.876798    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk"
Sep 17 14:19:54.950069 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:54.950038    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:55.159852 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:55.159762    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk"]
Sep 17 14:19:55.163088 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:19:55.163057    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00fdb0ea_d9f7_4dba_9e77_f26c85e50e0c.slice/crio-43d1f1e6deae43511776d607572270feb2ac99333fa41599b0a158b8bf8c6676 WatchSource:0}: Error finding container 43d1f1e6deae43511776d607572270feb2ac99333fa41599b0a158b8bf8c6676: Status 404 returned error can't find the container with id 43d1f1e6deae43511776d607572270feb2ac99333fa41599b0a158b8bf8c6676
Sep 17 14:19:55.165804 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:55.165763    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:19:55.165886 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:55.165868    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:19:55.165929 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:55.165916    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:19:55.507684 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:55.507590    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:55.663696 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:55.663659    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:55.954920 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:55.954883    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk" event={"ID":"00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c","Type":"ContainerStarted","Data":"e4ab3d7ece22c3f9c65ee25db7fa7a7d68653943f3699646e85483f25740a40b"}
Sep 17 14:19:55.955102 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:55.954940    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk" event={"ID":"00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c","Type":"ContainerStarted","Data":"43d1f1e6deae43511776d607572270feb2ac99333fa41599b0a158b8bf8c6676"}
Sep 17 14:19:55.983286 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:55.983206    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk" podStartSLOduration=1.983188613 podStartE2EDuration="1.983188613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:19:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:19:55.979053593 +0000 UTC m=+1464.921197238" watchObservedRunningTime="2026-09-17 14:19:55.983188613 +0000 UTC m=+1464.925332259"
Sep 17 14:19:56.549161 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:56.549125    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:56.838227 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:56.838130    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-079b38/e2e-conflict-b-9812c892-kserve-96d9c4849-ghjmf"
Sep 17 14:19:56.838227 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:56.838184    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-079b38/e2e-conflict-b-9812c892-kserve-96d9c4849-ghjmf"
Sep 17 14:19:56.850791 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:56.850763    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-079b38/e2e-conflict-b-9812c892-kserve-96d9c4849-ghjmf"
Sep 17 14:19:56.878426 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:56.878394    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk"
Sep 17 14:19:56.883577 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:56.883546    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk"
Sep 17 14:19:56.963110 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:56.963070    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk"
Sep 17 14:19:56.964056 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:56.964030    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk"
Sep 17 14:19:56.975017 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:56.974985    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-079b38/e2e-conflict-b-9812c892-kserve-96d9c4849-ghjmf"
Sep 17 14:19:57.509442 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:57.509406    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:57.902310 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:57.902268    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:58.002052 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:58.002022    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:58.454191 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:58.454150    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:59.558611 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:59.558569    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:59.659292 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:59.659257    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:59.764230 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:19:59.764195    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:20:00.194174 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:00.194142    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:20:00.292508 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:00.292449    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:20:00.833168 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:00.833137    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:20:01.897442 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:01.897399    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:20:02.948341 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:02.948304    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:20:03.858786 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:03.858754    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:20:04.779389 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:04.779347    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:20:05.527894 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:05.527853    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:20:10.630108 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:10.630073    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-5cdd59e7/e2e-lifecycle-model-5cdd59e7-kserve-655cf8b5dc-9rjlc"]
Sep 17 14:20:10.633900 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:10.633883    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5cdd59e7/e2e-lifecycle-model-5cdd59e7-kserve-655cf8b5dc-9rjlc"
Sep 17 14:20:10.637148 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:10.637123    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-5cdd59e7\"/\"e2e-lifecycle-model-5cdd59e7-kserve-self-signed-certs\""
Sep 17 14:20:10.637148 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:10.637140    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-5cdd59e7\"/\"kube-root-ca.crt\""
Sep 17 14:20:10.637331 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:10.637130    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-5cdd59e7\"/\"default-dockercfg-l67qv\""
Sep 17 14:20:10.637331 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:10.637179    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-5cdd59e7\"/\"openshift-service-ca.crt\""
Sep 17 14:20:10.644562 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:10.644540    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-5cdd59e7/e2e-lifecycle-model-5cdd59e7-kserve-655cf8b5dc-9rjlc"]
Sep 17 14:20:10.732448 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:10.732402    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:20:10.770724 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:10.770694    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/969c84d7-8c83-4483-ad5d-f52295b0dc2a-home\") pod \"e2e-lifecycle-model-5cdd59e7-kserve-655cf8b5dc-9rjlc\" (UID: \"969c84d7-8c83-4483-ad5d-f52295b0dc2a\") " pod="ai-tenant-e2e-mt-5cdd59e7/e2e-lifecycle-model-5cdd59e7-kserve-655cf8b5dc-9rjlc"
Sep 17 14:20:10.770852 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:10.770733    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/969c84d7-8c83-4483-ad5d-f52295b0dc2a-dshm\") pod \"e2e-lifecycle-model-5cdd59e7-kserve-655cf8b5dc-9rjlc\" (UID: \"969c84d7-8c83-4483-ad5d-f52295b0dc2a\") " pod="ai-tenant-e2e-mt-5cdd59e7/e2e-lifecycle-model-5cdd59e7-kserve-655cf8b5dc-9rjlc"
Sep 17 14:20:10.770894 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:10.770848    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/969c84d7-8c83-4483-ad5d-f52295b0dc2a-model-cache\") pod \"e2e-lifecycle-model-5cdd59e7-kserve-655cf8b5dc-9rjlc\" (UID: \"969c84d7-8c83-4483-ad5d-f52295b0dc2a\") " pod="ai-tenant-e2e-mt-5cdd59e7/e2e-lifecycle-model-5cdd59e7-kserve-655cf8b5dc-9rjlc"
Sep 17 14:20:10.770894 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:10.770884    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvtgp\" (UniqueName: \"kubernetes.io/projected/969c84d7-8c83-4483-ad5d-f52295b0dc2a-kube-api-access-cvtgp\") pod \"e2e-lifecycle-model-5cdd59e7-kserve-655cf8b5dc-9rjlc\" (UID: \"969c84d7-8c83-4483-ad5d-f52295b0dc2a\") " pod="ai-tenant-e2e-mt-5cdd59e7/e2e-lifecycle-model-5cdd59e7-kserve-655cf8b5dc-9rjlc"
Sep 17 14:20:10.770961 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:10.770906    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/969c84d7-8c83-4483-ad5d-f52295b0dc2a-tls-certs\") pod \"e2e-lifecycle-model-5cdd59e7-kserve-655cf8b5dc-9rjlc\" (UID: \"969c84d7-8c83-4483-ad5d-f52295b0dc2a\") " pod="ai-tenant-e2e-mt-5cdd59e7/e2e-lifecycle-model-5cdd59e7-kserve-655cf8b5dc-9rjlc"
Sep 17 14:20:10.770961 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:10.770926    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/969c84d7-8c83-4483-ad5d-f52295b0dc2a-tmp-dir\") pod \"e2e-lifecycle-model-5cdd59e7-kserve-655cf8b5dc-9rjlc\" (UID: \"969c84d7-8c83-4483-ad5d-f52295b0dc2a\") " pod="ai-tenant-e2e-mt-5cdd59e7/e2e-lifecycle-model-5cdd59e7-kserve-655cf8b5dc-9rjlc"
Sep 17 14:20:10.872368 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:10.872334    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/969c84d7-8c83-4483-ad5d-f52295b0dc2a-model-cache\") pod \"e2e-lifecycle-model-5cdd59e7-kserve-655cf8b5dc-9rjlc\" (UID: \"969c84d7-8c83-4483-ad5d-f52295b0dc2a\") " pod="ai-tenant-e2e-mt-5cdd59e7/e2e-lifecycle-model-5cdd59e7-kserve-655cf8b5dc-9rjlc"
Sep 17 14:20:10.872558 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:10.872380    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvtgp\" (UniqueName: \"kubernetes.io/projected/969c84d7-8c83-4483-ad5d-f52295b0dc2a-kube-api-access-cvtgp\") pod \"e2e-lifecycle-model-5cdd59e7-kserve-655cf8b5dc-9rjlc\" (UID: \"969c84d7-8c83-4483-ad5d-f52295b0dc2a\") " pod="ai-tenant-e2e-mt-5cdd59e7/e2e-lifecycle-model-5cdd59e7-kserve-655cf8b5dc-9rjlc"
Sep 17 14:20:10.872558 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:10.872509    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/969c84d7-8c83-4483-ad5d-f52295b0dc2a-tls-certs\") pod \"e2e-lifecycle-model-5cdd59e7-kserve-655cf8b5dc-9rjlc\" (UID: \"969c84d7-8c83-4483-ad5d-f52295b0dc2a\") " pod="ai-tenant-e2e-mt-5cdd59e7/e2e-lifecycle-model-5cdd59e7-kserve-655cf8b5dc-9rjlc"
Sep 17 14:20:10.872558 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:10.872552    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/969c84d7-8c83-4483-ad5d-f52295b0dc2a-tmp-dir\") pod \"e2e-lifecycle-model-5cdd59e7-kserve-655cf8b5dc-9rjlc\" (UID: \"969c84d7-8c83-4483-ad5d-f52295b0dc2a\") " pod="ai-tenant-e2e-mt-5cdd59e7/e2e-lifecycle-model-5cdd59e7-kserve-655cf8b5dc-9rjlc"
Sep 17 14:20:10.872720 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:10.872678    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/969c84d7-8c83-4483-ad5d-f52295b0dc2a-home\") pod \"e2e-lifecycle-model-5cdd59e7-kserve-655cf8b5dc-9rjlc\" (UID: \"969c84d7-8c83-4483-ad5d-f52295b0dc2a\") " pod="ai-tenant-e2e-mt-5cdd59e7/e2e-lifecycle-model-5cdd59e7-kserve-655cf8b5dc-9rjlc"
Sep 17 14:20:10.872782 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:10.872738    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/969c84d7-8c83-4483-ad5d-f52295b0dc2a-dshm\") pod \"e2e-lifecycle-model-5cdd59e7-kserve-655cf8b5dc-9rjlc\" (UID: \"969c84d7-8c83-4483-ad5d-f52295b0dc2a\") " pod="ai-tenant-e2e-mt-5cdd59e7/e2e-lifecycle-model-5cdd59e7-kserve-655cf8b5dc-9rjlc"
Sep 17 14:20:10.872837 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:10.872794    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/969c84d7-8c83-4483-ad5d-f52295b0dc2a-model-cache\") pod \"e2e-lifecycle-model-5cdd59e7-kserve-655cf8b5dc-9rjlc\" (UID: \"969c84d7-8c83-4483-ad5d-f52295b0dc2a\") " pod="ai-tenant-e2e-mt-5cdd59e7/e2e-lifecycle-model-5cdd59e7-kserve-655cf8b5dc-9rjlc"
Sep 17 14:20:10.872922 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:10.872903    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/969c84d7-8c83-4483-ad5d-f52295b0dc2a-tmp-dir\") pod \"e2e-lifecycle-model-5cdd59e7-kserve-655cf8b5dc-9rjlc\" (UID: \"969c84d7-8c83-4483-ad5d-f52295b0dc2a\") " pod="ai-tenant-e2e-mt-5cdd59e7/e2e-lifecycle-model-5cdd59e7-kserve-655cf8b5dc-9rjlc"
Sep 17 14:20:10.872993 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:10.872973    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/969c84d7-8c83-4483-ad5d-f52295b0dc2a-home\") pod \"e2e-lifecycle-model-5cdd59e7-kserve-655cf8b5dc-9rjlc\" (UID: \"969c84d7-8c83-4483-ad5d-f52295b0dc2a\") " pod="ai-tenant-e2e-mt-5cdd59e7/e2e-lifecycle-model-5cdd59e7-kserve-655cf8b5dc-9rjlc"
Sep 17 14:20:10.878300 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:10.878271    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/969c84d7-8c83-4483-ad5d-f52295b0dc2a-dshm\") pod \"e2e-lifecycle-model-5cdd59e7-kserve-655cf8b5dc-9rjlc\" (UID: \"969c84d7-8c83-4483-ad5d-f52295b0dc2a\") " pod="ai-tenant-e2e-mt-5cdd59e7/e2e-lifecycle-model-5cdd59e7-kserve-655cf8b5dc-9rjlc"
Sep 17 14:20:10.878877 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:10.878852    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/969c84d7-8c83-4483-ad5d-f52295b0dc2a-tls-certs\") pod \"e2e-lifecycle-model-5cdd59e7-kserve-655cf8b5dc-9rjlc\" (UID: \"969c84d7-8c83-4483-ad5d-f52295b0dc2a\") " pod="ai-tenant-e2e-mt-5cdd59e7/e2e-lifecycle-model-5cdd59e7-kserve-655cf8b5dc-9rjlc"
Sep 17 14:20:10.882376 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:10.881841    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvtgp\" (UniqueName: \"kubernetes.io/projected/969c84d7-8c83-4483-ad5d-f52295b0dc2a-kube-api-access-cvtgp\") pod \"e2e-lifecycle-model-5cdd59e7-kserve-655cf8b5dc-9rjlc\" (UID: \"969c84d7-8c83-4483-ad5d-f52295b0dc2a\") " pod="ai-tenant-e2e-mt-5cdd59e7/e2e-lifecycle-model-5cdd59e7-kserve-655cf8b5dc-9rjlc"
Sep 17 14:20:10.944768 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:10.944729    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5cdd59e7/e2e-lifecycle-model-5cdd59e7-kserve-655cf8b5dc-9rjlc"
Sep 17 14:20:11.221156 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:11.221127    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-5cdd59e7/e2e-lifecycle-model-5cdd59e7-kserve-655cf8b5dc-9rjlc"]
Sep 17 14:20:11.222019 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:20:11.221990    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod969c84d7_8c83_4483_ad5d_f52295b0dc2a.slice/crio-7e0a93d9f79c11a6fcda5dec8778b8a3ac72c833294275302644f7978525bc79 WatchSource:0}: Error finding container 7e0a93d9f79c11a6fcda5dec8778b8a3ac72c833294275302644f7978525bc79: Status 404 returned error can't find the container with id 7e0a93d9f79c11a6fcda5dec8778b8a3ac72c833294275302644f7978525bc79
Sep 17 14:20:12.070618 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:12.070571    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5cdd59e7/e2e-lifecycle-model-5cdd59e7-kserve-655cf8b5dc-9rjlc" event={"ID":"969c84d7-8c83-4483-ad5d-f52295b0dc2a","Type":"ContainerStarted","Data":"06003c7d33d74ed284a3377baccd95e48376aab7c16a3e3a2f66fa55fda6c943"}
Sep 17 14:20:12.071142 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:12.070631    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5cdd59e7/e2e-lifecycle-model-5cdd59e7-kserve-655cf8b5dc-9rjlc" event={"ID":"969c84d7-8c83-4483-ad5d-f52295b0dc2a","Type":"ContainerStarted","Data":"7e0a93d9f79c11a6fcda5dec8778b8a3ac72c833294275302644f7978525bc79"}
Sep 17 14:20:12.097690 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:12.097626    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-5cdd59e7/e2e-lifecycle-model-5cdd59e7-kserve-655cf8b5dc-9rjlc" podStartSLOduration=2.097603552 podStartE2EDuration="2.097603552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:20:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:20:12.092980339 +0000 UTC m=+1481.035124045" watchObservedRunningTime="2026-09-17 14:20:12.097603552 +0000 UTC m=+1481.039747200"
Sep 17 14:20:20.945881 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:20.945843    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-5cdd59e7/e2e-lifecycle-model-5cdd59e7-kserve-655cf8b5dc-9rjlc"
Sep 17 14:20:20.946308 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:20.945898    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-5cdd59e7/e2e-lifecycle-model-5cdd59e7-kserve-655cf8b5dc-9rjlc"
Sep 17 14:20:20.958533 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:20.958510    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-5cdd59e7/e2e-lifecycle-model-5cdd59e7-kserve-655cf8b5dc-9rjlc"
Sep 17 14:20:21.142347 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:21.142315    2560 generic.go:356] "Generic (PLEG): container finished" podID="00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c" containerID="e4ab3d7ece22c3f9c65ee25db7fa7a7d68653943f3699646e85483f25740a40b" exitCode=0
Sep 17 14:20:21.142551 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:21.142390    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk" event={"ID":"00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c","Type":"ContainerDied","Data":"e4ab3d7ece22c3f9c65ee25db7fa7a7d68653943f3699646e85483f25740a40b"}
Sep 17 14:20:21.142872 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:21.142852    2560 scope.go:117] "RemoveContainer" containerID="e4ab3d7ece22c3f9c65ee25db7fa7a7d68653943f3699646e85483f25740a40b"
Sep 17 14:20:21.144249 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:21.144218    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:20:21.144343 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:21.144307    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:20:21.144395 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:21.144354    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:20:21.160069 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:21.160029    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-5cdd59e7/e2e-lifecycle-model-5cdd59e7-kserve-655cf8b5dc-9rjlc"
Sep 17 14:20:22.041660 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:22.041599    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:20:22.151494 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:22.151434    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk" event={"ID":"00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c","Type":"ContainerStarted","Data":"37df4fdacbf70607e486aa4270c16db4ff2fba4c42bc6c2034ecc83a72fbe048"}
Sep 17 14:20:22.871605 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:22.871567    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:20:22.877851 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:22.877824    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk"
Sep 17 14:20:22.879409 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:22.879381    2560 patch_prober.go:28] interesting pod/e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Sep 17 14:20:22.879678 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:22.879648    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk" podUID="00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Sep 17 14:20:22.968299 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:22.968256    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:20:23.877781 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:23.877734    2560 patch_prober.go:28] interesting pod/e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Sep 17 14:20:23.878295 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:23.877805    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk" podUID="00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Sep 17 14:20:23.898846 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:23.898812    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:20:24.878220 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:24.878170    2560 patch_prober.go:28] interesting pod/e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Sep 17 14:20:24.878749 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:24.878250    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk" podUID="00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Sep 17 14:20:24.947891 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:24.947856    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:20:25.869430 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:25.869387    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:20:25.878164 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:25.878123    2560 patch_prober.go:28] interesting pod/e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Sep 17 14:20:25.878376 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:25.878203    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk" podUID="00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Sep 17 14:20:26.647324 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:26.647278    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:20:26.878573 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:26.878532    2560 patch_prober.go:28] interesting pod/e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Sep 17 14:20:26.879107 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:26.878605    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk" podUID="00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Sep 17 14:20:26.963830 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:26.963725    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk"
Sep 17 14:20:27.548929 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:27.548894    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:20:27.882744 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:27.882669    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk"
Sep 17 14:20:27.883804 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:27.883781    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk"
Sep 17 14:20:28.295324 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:28.295224    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:20:31.808846 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:31.808807    2560 scope.go:117] "RemoveContainer" containerID="ccac192eab7db273755b327d17ae27f6cd518d4b4564b93ce82aa938797d14a3"
Sep 17 14:20:31.819244 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:31.819219    2560 scope.go:117] "RemoveContainer" containerID="6cad336385e879ea9f08ec96cea3193f66ed397130f121a7db80342181dd33eb"
Sep 17 14:20:55.663604 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:55.663517    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:20:56.977502 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:56.977455    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:20:57.078904 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:20:57.078867    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:21:00.443177 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:00.443138    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:21:03.047644 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:03.047612    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-079b38/e2e-conflict-b-9812c892-kserve-96d9c4849-ghjmf"]
Sep 17 14:21:03.104518 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:03.104456    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:21:12.859328 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:12.859289    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cdd59e7-5lrd9"]
Sep 17 14:21:12.863073 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:12.863052    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cdd59e7-5lrd9"
Sep 17 14:21:12.865289 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:12.865268    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 14:21:12.865385 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:12.865364    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rp9ww\""
Sep 17 14:21:12.872994 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:12.872969    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cdd59e7-5lrd9"]
Sep 17 14:21:12.888900 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:12.888877    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54f05934-ea3c-4731-8918-2436f7c97c9e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-5cdd59e7-5lrd9\" (UID: \"54f05934-ea3c-4731-8918-2436f7c97c9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cdd59e7-5lrd9"
Sep 17 14:21:12.989931 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:12.989902    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54f05934-ea3c-4731-8918-2436f7c97c9e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-5cdd59e7-5lrd9\" (UID: \"54f05934-ea3c-4731-8918-2436f7c97c9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cdd59e7-5lrd9"
Sep 17 14:21:12.990572 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:12.990551    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54f05934-ea3c-4731-8918-2436f7c97c9e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-5cdd59e7-5lrd9\" (UID: \"54f05934-ea3c-4731-8918-2436f7c97c9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cdd59e7-5lrd9"
Sep 17 14:21:13.143572 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:13.143477    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:21:13.175506 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:13.175454    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cdd59e7-5lrd9"
Sep 17 14:21:13.453177 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:13.453145    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cdd59e7-5lrd9"]
Sep 17 14:21:13.453867 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:21:13.453842    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54f05934_ea3c_4731_8918_2436f7c97c9e.slice/crio-ec2305993150dcd9ba8847d1238a110f025692cd747c5ea48dda630e0e24aa31 WatchSource:0}: Error finding container ec2305993150dcd9ba8847d1238a110f025692cd747c5ea48dda630e0e24aa31: Status 404 returned error can't find the container with id ec2305993150dcd9ba8847d1238a110f025692cd747c5ea48dda630e0e24aa31
Sep 17 14:21:13.510548 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:13.510513    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cdd59e7-5lrd9" event={"ID":"54f05934-ea3c-4731-8918-2436f7c97c9e","Type":"ContainerStarted","Data":"ec2305993150dcd9ba8847d1238a110f025692cd747c5ea48dda630e0e24aa31"}
Sep 17 14:21:13.937044 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:13.937016    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:21:14.518978 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:14.518891    2560 generic.go:356] "Generic (PLEG): container finished" podID="54f05934-ea3c-4731-8918-2436f7c97c9e" containerID="b238172f01a8ee478f30d15e87d63c05ccf532d2effe7cad7422e1f4f3e2c892" exitCode=0
Sep 17 14:21:14.518978 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:14.518944    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cdd59e7-5lrd9" event={"ID":"54f05934-ea3c-4731-8918-2436f7c97c9e","Type":"ContainerDied","Data":"b238172f01a8ee478f30d15e87d63c05ccf532d2effe7cad7422e1f4f3e2c892"}
Sep 17 14:21:14.860902 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:14.860865    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:21:15.297334 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:15.297245    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-079b38/e2e-conflict-a-9812c892-kserve-5655867767-qkchj"]
Sep 17 14:21:15.312611 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:15.312582    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:21:15.651431 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:15.651410    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cdd59e7-5lrd9"
Sep 17 14:21:15.654037 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:15.654008    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:21:15.720047 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:15.720014    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54f05934-ea3c-4731-8918-2436f7c97c9e-maas-api-service-ca\") pod \"54f05934-ea3c-4731-8918-2436f7c97c9e\" (UID: \"54f05934-ea3c-4731-8918-2436f7c97c9e\") "
Sep 17 14:21:15.720383 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:15.720360    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54f05934-ea3c-4731-8918-2436f7c97c9e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "54f05934-ea3c-4731-8918-2436f7c97c9e" (UID: "54f05934-ea3c-4731-8918-2436f7c97c9e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:21:15.720576 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:15.720559    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54f05934-ea3c-4731-8918-2436f7c97c9e-maas-api-service-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:21:16.540448 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:16.540408    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:21:16.541226 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:16.541208    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cdd59e7-5lrd9"
Sep 17 14:21:16.541331 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:16.541245    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cdd59e7-5lrd9" event={"ID":"54f05934-ea3c-4731-8918-2436f7c97c9e","Type":"ContainerDied","Data":"ec2305993150dcd9ba8847d1238a110f025692cd747c5ea48dda630e0e24aa31"}
Sep 17 14:21:16.541331 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:16.541314    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec2305993150dcd9ba8847d1238a110f025692cd747c5ea48dda630e0e24aa31"
Sep 17 14:21:18.015370 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:18.015335    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:21:18.066849 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:18.066805    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-079b38/e2e-conflict-b-9812c892-kserve-96d9c4849-ghjmf" podUID="3ea78297-9a5f-4d0c-9912-a4fd415043ec" containerName="main" containerID="cri-o://5538cfd0116c172add529ba3fe0797fd7ad84ac5cdc4120a8ff62c65258e6f38" gracePeriod=45
Sep 17 14:21:18.326131 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:18.326105    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-079b38/e2e-conflict-b-9812c892-kserve-96d9c4849-ghjmf"
Sep 17 14:21:18.354078 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:18.354051    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ea78297-9a5f-4d0c-9912-a4fd415043ec-tls-certs\") pod \"3ea78297-9a5f-4d0c-9912-a4fd415043ec\" (UID: \"3ea78297-9a5f-4d0c-9912-a4fd415043ec\") "
Sep 17 14:21:18.354213 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:18.354119    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ea78297-9a5f-4d0c-9912-a4fd415043ec-tmp-dir\") pod \"3ea78297-9a5f-4d0c-9912-a4fd415043ec\" (UID: \"3ea78297-9a5f-4d0c-9912-a4fd415043ec\") "
Sep 17 14:21:18.354213 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:18.354140    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ea78297-9a5f-4d0c-9912-a4fd415043ec-model-cache\") pod \"3ea78297-9a5f-4d0c-9912-a4fd415043ec\" (UID: \"3ea78297-9a5f-4d0c-9912-a4fd415043ec\") "
Sep 17 14:21:18.354213 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:18.354175    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jslws\" (UniqueName: \"kubernetes.io/projected/3ea78297-9a5f-4d0c-9912-a4fd415043ec-kube-api-access-jslws\") pod \"3ea78297-9a5f-4d0c-9912-a4fd415043ec\" (UID: \"3ea78297-9a5f-4d0c-9912-a4fd415043ec\") "
Sep 17 14:21:18.354405 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:18.354259    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ea78297-9a5f-4d0c-9912-a4fd415043ec-home\") pod \"3ea78297-9a5f-4d0c-9912-a4fd415043ec\" (UID: \"3ea78297-9a5f-4d0c-9912-a4fd415043ec\") "
Sep 17 14:21:18.354405 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:18.354298    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ea78297-9a5f-4d0c-9912-a4fd415043ec-dshm\") pod \"3ea78297-9a5f-4d0c-9912-a4fd415043ec\" (UID: \"3ea78297-9a5f-4d0c-9912-a4fd415043ec\") "
Sep 17 14:21:18.354547 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:18.354444    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ea78297-9a5f-4d0c-9912-a4fd415043ec-model-cache" (OuterVolumeSpecName: "model-cache") pod "3ea78297-9a5f-4d0c-9912-a4fd415043ec" (UID: "3ea78297-9a5f-4d0c-9912-a4fd415043ec"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:21:18.354760 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:18.354729    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ea78297-9a5f-4d0c-9912-a4fd415043ec-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3ea78297-9a5f-4d0c-9912-a4fd415043ec" (UID: "3ea78297-9a5f-4d0c-9912-a4fd415043ec"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:21:18.354879 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:18.354863    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ea78297-9a5f-4d0c-9912-a4fd415043ec-tmp-dir\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:21:18.354879 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:18.354861    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ea78297-9a5f-4d0c-9912-a4fd415043ec-home" (OuterVolumeSpecName: "home") pod "3ea78297-9a5f-4d0c-9912-a4fd415043ec" (UID: "3ea78297-9a5f-4d0c-9912-a4fd415043ec"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:21:18.355010 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:18.354882    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ea78297-9a5f-4d0c-9912-a4fd415043ec-model-cache\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:21:18.356915 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:18.356890    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ea78297-9a5f-4d0c-9912-a4fd415043ec-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3ea78297-9a5f-4d0c-9912-a4fd415043ec" (UID: "3ea78297-9a5f-4d0c-9912-a4fd415043ec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:21:18.357007 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:18.356909    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ea78297-9a5f-4d0c-9912-a4fd415043ec-dshm" (OuterVolumeSpecName: "dshm") pod "3ea78297-9a5f-4d0c-9912-a4fd415043ec" (UID: "3ea78297-9a5f-4d0c-9912-a4fd415043ec"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:21:18.357067 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:18.357048    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ea78297-9a5f-4d0c-9912-a4fd415043ec-kube-api-access-jslws" (OuterVolumeSpecName: "kube-api-access-jslws") pod "3ea78297-9a5f-4d0c-9912-a4fd415043ec" (UID: "3ea78297-9a5f-4d0c-9912-a4fd415043ec"). InnerVolumeSpecName "kube-api-access-jslws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:21:18.455802 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:18.455769    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ea78297-9a5f-4d0c-9912-a4fd415043ec-home\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:21:18.455802 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:18.455795    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ea78297-9a5f-4d0c-9912-a4fd415043ec-dshm\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:21:18.455802 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:18.455804    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ea78297-9a5f-4d0c-9912-a4fd415043ec-tls-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:21:18.455802 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:18.455814    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jslws\" (UniqueName: \"kubernetes.io/projected/3ea78297-9a5f-4d0c-9912-a4fd415043ec-kube-api-access-jslws\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:21:18.555857 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:18.555756    2560 generic.go:356] "Generic (PLEG): container finished" podID="3ea78297-9a5f-4d0c-9912-a4fd415043ec" containerID="5538cfd0116c172add529ba3fe0797fd7ad84ac5cdc4120a8ff62c65258e6f38" exitCode=0
Sep 17 14:21:18.555857 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:18.555808    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-079b38/e2e-conflict-b-9812c892-kserve-96d9c4849-ghjmf"
Sep 17 14:21:18.556088 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:18.555849    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-079b38/e2e-conflict-b-9812c892-kserve-96d9c4849-ghjmf" event={"ID":"3ea78297-9a5f-4d0c-9912-a4fd415043ec","Type":"ContainerDied","Data":"5538cfd0116c172add529ba3fe0797fd7ad84ac5cdc4120a8ff62c65258e6f38"}
Sep 17 14:21:18.556088 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:18.555920    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-079b38/e2e-conflict-b-9812c892-kserve-96d9c4849-ghjmf" event={"ID":"3ea78297-9a5f-4d0c-9912-a4fd415043ec","Type":"ContainerDied","Data":"bb31739b481e0eaa25cd4d246d4ea660c0b17855ffa74d12ec6bac038b5daa57"}
Sep 17 14:21:18.556088 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:18.555941    2560 scope.go:117] "RemoveContainer" containerID="5538cfd0116c172add529ba3fe0797fd7ad84ac5cdc4120a8ff62c65258e6f38"
Sep 17 14:21:18.569070 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:18.569032    2560 scope.go:117] "RemoveContainer" containerID="5538cfd0116c172add529ba3fe0797fd7ad84ac5cdc4120a8ff62c65258e6f38"
Sep 17 14:21:18.569498 ip-10-0-129-158 kubenswrapper[2560]: E0917 14:21:18.569442    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5538cfd0116c172add529ba3fe0797fd7ad84ac5cdc4120a8ff62c65258e6f38\": container with ID starting with 5538cfd0116c172add529ba3fe0797fd7ad84ac5cdc4120a8ff62c65258e6f38 not found: ID does not exist" containerID="5538cfd0116c172add529ba3fe0797fd7ad84ac5cdc4120a8ff62c65258e6f38"
Sep 17 14:21:18.569601 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:18.569507    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5538cfd0116c172add529ba3fe0797fd7ad84ac5cdc4120a8ff62c65258e6f38"} err="failed to get container status \"5538cfd0116c172add529ba3fe0797fd7ad84ac5cdc4120a8ff62c65258e6f38\": rpc error: code = NotFound desc = could not find container \"5538cfd0116c172add529ba3fe0797fd7ad84ac5cdc4120a8ff62c65258e6f38\": container with ID starting with 5538cfd0116c172add529ba3fe0797fd7ad84ac5cdc4120a8ff62c65258e6f38 not found: ID does not exist"
Sep 17 14:21:18.591815 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:18.591787    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-079b38/e2e-conflict-b-9812c892-kserve-96d9c4849-ghjmf"]
Sep 17 14:21:18.598928 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:18.598903    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-079b38/e2e-conflict-b-9812c892-kserve-96d9c4849-ghjmf"]
Sep 17 14:21:18.772023 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:18.771986    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:21:19.377127 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:19.377092    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:21:19.570331 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:19.570297    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ea78297-9a5f-4d0c-9912-a4fd415043ec" path="/var/lib/kubelet/pods/3ea78297-9a5f-4d0c-9912-a4fd415043ec/volumes"
Sep 17 14:21:20.261095 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:20.261061    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:21:26.439809 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:26.439768    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-079b38/distinct-w4-079b38-kserve-7fd69fb686-5drgf"]
Sep 17 14:21:26.440562 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:26.440540    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54f05934-ea3c-4731-8918-2436f7c97c9e" containerName="revoke-keys"
Sep 17 14:21:26.440618 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:26.440565    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="54f05934-ea3c-4731-8918-2436f7c97c9e" containerName="revoke-keys"
Sep 17 14:21:26.440618 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:26.440588    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ea78297-9a5f-4d0c-9912-a4fd415043ec" containerName="main"
Sep 17 14:21:26.440618 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:26.440597    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ea78297-9a5f-4d0c-9912-a4fd415043ec" containerName="main"
Sep 17 14:21:26.440763 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:26.440748    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ea78297-9a5f-4d0c-9912-a4fd415043ec" containerName="main"
Sep 17 14:21:26.440798 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:26.440768    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="54f05934-ea3c-4731-8918-2436f7c97c9e" containerName="revoke-keys"
Sep 17 14:21:26.446025 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:26.445997    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-079b38/distinct-w4-079b38-kserve-7fd69fb686-5drgf"
Sep 17 14:21:26.448671 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:26.448645    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-079b38\"/\"distinct-w4-079b38-kserve-self-signed-certs\""
Sep 17 14:21:26.457401 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:26.457375    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-079b38/distinct-w4-079b38-kserve-7fd69fb686-5drgf"]
Sep 17 14:21:26.486222 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:26.486189    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:21:26.543729 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:26.543692    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxsmz\" (UniqueName: \"kubernetes.io/projected/0cd1d4fd-bb83-4910-89b7-191b012d9268-kube-api-access-kxsmz\") pod \"distinct-w4-079b38-kserve-7fd69fb686-5drgf\" (UID: \"0cd1d4fd-bb83-4910-89b7-191b012d9268\") " pod="e2e-models-e2e-worker-w4-079b38/distinct-w4-079b38-kserve-7fd69fb686-5drgf"
Sep 17 14:21:26.543903 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:26.543736    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cd1d4fd-bb83-4910-89b7-191b012d9268-model-cache\") pod \"distinct-w4-079b38-kserve-7fd69fb686-5drgf\" (UID: \"0cd1d4fd-bb83-4910-89b7-191b012d9268\") " pod="e2e-models-e2e-worker-w4-079b38/distinct-w4-079b38-kserve-7fd69fb686-5drgf"
Sep 17 14:21:26.543903 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:26.543788    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cd1d4fd-bb83-4910-89b7-191b012d9268-tls-certs\") pod \"distinct-w4-079b38-kserve-7fd69fb686-5drgf\" (UID: \"0cd1d4fd-bb83-4910-89b7-191b012d9268\") " pod="e2e-models-e2e-worker-w4-079b38/distinct-w4-079b38-kserve-7fd69fb686-5drgf"
Sep 17 14:21:26.543903 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:26.543834    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cd1d4fd-bb83-4910-89b7-191b012d9268-dshm\") pod \"distinct-w4-079b38-kserve-7fd69fb686-5drgf\" (UID: \"0cd1d4fd-bb83-4910-89b7-191b012d9268\") " pod="e2e-models-e2e-worker-w4-079b38/distinct-w4-079b38-kserve-7fd69fb686-5drgf"
Sep 17 14:21:26.544012 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:26.543929    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cd1d4fd-bb83-4910-89b7-191b012d9268-tmp-dir\") pod \"distinct-w4-079b38-kserve-7fd69fb686-5drgf\" (UID: \"0cd1d4fd-bb83-4910-89b7-191b012d9268\") " pod="e2e-models-e2e-worker-w4-079b38/distinct-w4-079b38-kserve-7fd69fb686-5drgf"
Sep 17 14:21:26.544012 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:26.543990    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cd1d4fd-bb83-4910-89b7-191b012d9268-home\") pod \"distinct-w4-079b38-kserve-7fd69fb686-5drgf\" (UID: \"0cd1d4fd-bb83-4910-89b7-191b012d9268\") " pod="e2e-models-e2e-worker-w4-079b38/distinct-w4-079b38-kserve-7fd69fb686-5drgf"
Sep 17 14:21:26.645530 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:26.645495    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cd1d4fd-bb83-4910-89b7-191b012d9268-dshm\") pod \"distinct-w4-079b38-kserve-7fd69fb686-5drgf\" (UID: \"0cd1d4fd-bb83-4910-89b7-191b012d9268\") " pod="e2e-models-e2e-worker-w4-079b38/distinct-w4-079b38-kserve-7fd69fb686-5drgf"
Sep 17 14:21:26.645707 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:26.645565    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cd1d4fd-bb83-4910-89b7-191b012d9268-tmp-dir\") pod \"distinct-w4-079b38-kserve-7fd69fb686-5drgf\" (UID: \"0cd1d4fd-bb83-4910-89b7-191b012d9268\") " pod="e2e-models-e2e-worker-w4-079b38/distinct-w4-079b38-kserve-7fd69fb686-5drgf"
Sep 17 14:21:26.645707 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:26.645696    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cd1d4fd-bb83-4910-89b7-191b012d9268-home\") pod \"distinct-w4-079b38-kserve-7fd69fb686-5drgf\" (UID: \"0cd1d4fd-bb83-4910-89b7-191b012d9268\") " pod="e2e-models-e2e-worker-w4-079b38/distinct-w4-079b38-kserve-7fd69fb686-5drgf"
Sep 17 14:21:26.645884 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:26.645865    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxsmz\" (UniqueName: \"kubernetes.io/projected/0cd1d4fd-bb83-4910-89b7-191b012d9268-kube-api-access-kxsmz\") pod \"distinct-w4-079b38-kserve-7fd69fb686-5drgf\" (UID: \"0cd1d4fd-bb83-4910-89b7-191b012d9268\") " pod="e2e-models-e2e-worker-w4-079b38/distinct-w4-079b38-kserve-7fd69fb686-5drgf"
Sep 17 14:21:26.645944 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:26.645898    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cd1d4fd-bb83-4910-89b7-191b012d9268-model-cache\") pod \"distinct-w4-079b38-kserve-7fd69fb686-5drgf\" (UID: \"0cd1d4fd-bb83-4910-89b7-191b012d9268\") " pod="e2e-models-e2e-worker-w4-079b38/distinct-w4-079b38-kserve-7fd69fb686-5drgf"
Sep 17 14:21:26.646000 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:26.645947    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cd1d4fd-bb83-4910-89b7-191b012d9268-tls-certs\") pod \"distinct-w4-079b38-kserve-7fd69fb686-5drgf\" (UID: \"0cd1d4fd-bb83-4910-89b7-191b012d9268\") " pod="e2e-models-e2e-worker-w4-079b38/distinct-w4-079b38-kserve-7fd69fb686-5drgf"
Sep 17 14:21:26.646054 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:26.646026    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cd1d4fd-bb83-4910-89b7-191b012d9268-tmp-dir\") pod \"distinct-w4-079b38-kserve-7fd69fb686-5drgf\" (UID: \"0cd1d4fd-bb83-4910-89b7-191b012d9268\") " pod="e2e-models-e2e-worker-w4-079b38/distinct-w4-079b38-kserve-7fd69fb686-5drgf"
Sep 17 14:21:26.646115 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:26.646068    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cd1d4fd-bb83-4910-89b7-191b012d9268-home\") pod \"distinct-w4-079b38-kserve-7fd69fb686-5drgf\" (UID: \"0cd1d4fd-bb83-4910-89b7-191b012d9268\") " pod="e2e-models-e2e-worker-w4-079b38/distinct-w4-079b38-kserve-7fd69fb686-5drgf"
Sep 17 14:21:26.646269 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:26.646248    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cd1d4fd-bb83-4910-89b7-191b012d9268-model-cache\") pod \"distinct-w4-079b38-kserve-7fd69fb686-5drgf\" (UID: \"0cd1d4fd-bb83-4910-89b7-191b012d9268\") " pod="e2e-models-e2e-worker-w4-079b38/distinct-w4-079b38-kserve-7fd69fb686-5drgf"
Sep 17 14:21:26.647764 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:26.647738    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cd1d4fd-bb83-4910-89b7-191b012d9268-dshm\") pod \"distinct-w4-079b38-kserve-7fd69fb686-5drgf\" (UID: \"0cd1d4fd-bb83-4910-89b7-191b012d9268\") " pod="e2e-models-e2e-worker-w4-079b38/distinct-w4-079b38-kserve-7fd69fb686-5drgf"
Sep 17 14:21:26.648182 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:26.648166    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cd1d4fd-bb83-4910-89b7-191b012d9268-tls-certs\") pod \"distinct-w4-079b38-kserve-7fd69fb686-5drgf\" (UID: \"0cd1d4fd-bb83-4910-89b7-191b012d9268\") " pod="e2e-models-e2e-worker-w4-079b38/distinct-w4-079b38-kserve-7fd69fb686-5drgf"
Sep 17 14:21:26.654995 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:26.654968    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxsmz\" (UniqueName: \"kubernetes.io/projected/0cd1d4fd-bb83-4910-89b7-191b012d9268-kube-api-access-kxsmz\") pod \"distinct-w4-079b38-kserve-7fd69fb686-5drgf\" (UID: \"0cd1d4fd-bb83-4910-89b7-191b012d9268\") " pod="e2e-models-e2e-worker-w4-079b38/distinct-w4-079b38-kserve-7fd69fb686-5drgf"
Sep 17 14:21:26.757200 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:26.757136    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-079b38/distinct-w4-079b38-kserve-7fd69fb686-5drgf"
Sep 17 14:21:27.033451 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:27.033419    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-079b38/distinct-w4-079b38-kserve-7fd69fb686-5drgf"]
Sep 17 14:21:27.034421 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:21:27.034394    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0cd1d4fd_bb83_4910_89b7_191b012d9268.slice/crio-7b54f2bc78b1a2a26782a76df7f616d2775111c24f73b3d8d3b640edd7c76b48 WatchSource:0}: Error finding container 7b54f2bc78b1a2a26782a76df7f616d2775111c24f73b3d8d3b640edd7c76b48: Status 404 returned error can't find the container with id 7b54f2bc78b1a2a26782a76df7f616d2775111c24f73b3d8d3b640edd7c76b48
Sep 17 14:21:27.625746 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:27.625709    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-079b38/distinct-w4-079b38-kserve-7fd69fb686-5drgf" event={"ID":"0cd1d4fd-bb83-4910-89b7-191b012d9268","Type":"ContainerStarted","Data":"c471354c7271d61d050f2f00da3c6da7795d1ff59a7c267ff0d8cd8b8efb90e7"}
Sep 17 14:21:27.625746 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:27.625751    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-079b38/distinct-w4-079b38-kserve-7fd69fb686-5drgf" event={"ID":"0cd1d4fd-bb83-4910-89b7-191b012d9268","Type":"ContainerStarted","Data":"7b54f2bc78b1a2a26782a76df7f616d2775111c24f73b3d8d3b640edd7c76b48"}
Sep 17 14:21:27.654020 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:27.653971    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-079b38/distinct-w4-079b38-kserve-7fd69fb686-5drgf" podStartSLOduration=1.653956618 podStartE2EDuration="1.653956618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:21:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:21:27.650762126 +0000 UTC m=+1556.592905794" watchObservedRunningTime="2026-09-17 14:21:27.653956618 +0000 UTC m=+1556.596100264"
Sep 17 14:21:29.050483 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:29.050302    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cdd59e7-5lrd9"]
Sep 17 14:21:29.055590 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:29.055499    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5cdd59e7-5lrd9"]
Sep 17 14:21:29.519500 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:29.519396    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:21:29.570130 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:29.570098    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54f05934-ea3c-4731-8918-2436f7c97c9e" path="/var/lib/kubelet/pods/54f05934-ea3c-4731-8918-2436f7c97c9e/volumes"
Sep 17 14:21:29.617963 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:29.617926    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:21:30.303177 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:30.303143    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:21:30.316840 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:30.316812    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-079b38/e2e-conflict-a-9812c892-kserve-5655867767-qkchj" podUID="6e903702-6868-403e-aabb-94e3faac371e" containerName="main" containerID="cri-o://4c59fb512c905c00ee41a5e8012c7107e56d3819df79b184c1358d5dc31e85c1" gracePeriod=45
Sep 17 14:21:30.404293 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:30.404249    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:21:30.563124 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:30.563067    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-079b38/e2e-conflict-a-9812c892-kserve-5655867767-qkchj"
Sep 17 14:21:30.629359 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:30.629306    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:21:30.647438 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:30.647414    2560 generic.go:356] "Generic (PLEG): container finished" podID="6e903702-6868-403e-aabb-94e3faac371e" containerID="4c59fb512c905c00ee41a5e8012c7107e56d3819df79b184c1358d5dc31e85c1" exitCode=0
Sep 17 14:21:30.647587 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:30.647451    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-079b38/e2e-conflict-a-9812c892-kserve-5655867767-qkchj"
Sep 17 14:21:30.647587 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:30.647496    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-079b38/e2e-conflict-a-9812c892-kserve-5655867767-qkchj" event={"ID":"6e903702-6868-403e-aabb-94e3faac371e","Type":"ContainerDied","Data":"4c59fb512c905c00ee41a5e8012c7107e56d3819df79b184c1358d5dc31e85c1"}
Sep 17 14:21:30.647587 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:30.647536    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-079b38/e2e-conflict-a-9812c892-kserve-5655867767-qkchj" event={"ID":"6e903702-6868-403e-aabb-94e3faac371e","Type":"ContainerDied","Data":"278b0805148f69df0feeb6a060132ad2b223fb81956f284d283cc669935204a5"}
Sep 17 14:21:30.647587 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:30.647554    2560 scope.go:117] "RemoveContainer" containerID="4c59fb512c905c00ee41a5e8012c7107e56d3819df79b184c1358d5dc31e85c1"
Sep 17 14:21:30.656913 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:30.656899    2560 scope.go:117] "RemoveContainer" containerID="4c59fb512c905c00ee41a5e8012c7107e56d3819df79b184c1358d5dc31e85c1"
Sep 17 14:21:30.657179 ip-10-0-129-158 kubenswrapper[2560]: E0917 14:21:30.657164    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c59fb512c905c00ee41a5e8012c7107e56d3819df79b184c1358d5dc31e85c1\": container with ID starting with 4c59fb512c905c00ee41a5e8012c7107e56d3819df79b184c1358d5dc31e85c1 not found: ID does not exist" containerID="4c59fb512c905c00ee41a5e8012c7107e56d3819df79b184c1358d5dc31e85c1"
Sep 17 14:21:30.657215 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:30.657188    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c59fb512c905c00ee41a5e8012c7107e56d3819df79b184c1358d5dc31e85c1"} err="failed to get container status \"4c59fb512c905c00ee41a5e8012c7107e56d3819df79b184c1358d5dc31e85c1\": rpc error: code = NotFound desc = could not find container \"4c59fb512c905c00ee41a5e8012c7107e56d3819df79b184c1358d5dc31e85c1\": container with ID starting with 4c59fb512c905c00ee41a5e8012c7107e56d3819df79b184c1358d5dc31e85c1 not found: ID does not exist"
Sep 17 14:21:30.688078 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:30.688054    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e903702-6868-403e-aabb-94e3faac371e-dshm\") pod \"6e903702-6868-403e-aabb-94e3faac371e\" (UID: \"6e903702-6868-403e-aabb-94e3faac371e\") "
Sep 17 14:21:30.688173 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:30.688110    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e903702-6868-403e-aabb-94e3faac371e-tmp-dir\") pod \"6e903702-6868-403e-aabb-94e3faac371e\" (UID: \"6e903702-6868-403e-aabb-94e3faac371e\") "
Sep 17 14:21:30.688173 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:30.688141    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e903702-6868-403e-aabb-94e3faac371e-tls-certs\") pod \"6e903702-6868-403e-aabb-94e3faac371e\" (UID: \"6e903702-6868-403e-aabb-94e3faac371e\") "
Sep 17 14:21:30.688251 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:30.688236    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e903702-6868-403e-aabb-94e3faac371e-home\") pod \"6e903702-6868-403e-aabb-94e3faac371e\" (UID: \"6e903702-6868-403e-aabb-94e3faac371e\") "
Sep 17 14:21:30.688298 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:30.688255    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e903702-6868-403e-aabb-94e3faac371e-model-cache\") pod \"6e903702-6868-403e-aabb-94e3faac371e\" (UID: \"6e903702-6868-403e-aabb-94e3faac371e\") "
Sep 17 14:21:30.688298 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:30.688284    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l4m9c\" (UniqueName: \"kubernetes.io/projected/6e903702-6868-403e-aabb-94e3faac371e-kube-api-access-l4m9c\") pod \"6e903702-6868-403e-aabb-94e3faac371e\" (UID: \"6e903702-6868-403e-aabb-94e3faac371e\") "
Sep 17 14:21:30.688435 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:30.688405    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e903702-6868-403e-aabb-94e3faac371e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6e903702-6868-403e-aabb-94e3faac371e" (UID: "6e903702-6868-403e-aabb-94e3faac371e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:21:30.688545 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:30.688525    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e903702-6868-403e-aabb-94e3faac371e-model-cache" (OuterVolumeSpecName: "model-cache") pod "6e903702-6868-403e-aabb-94e3faac371e" (UID: "6e903702-6868-403e-aabb-94e3faac371e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:21:30.688612 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:30.688541    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e903702-6868-403e-aabb-94e3faac371e-home" (OuterVolumeSpecName: "home") pod "6e903702-6868-403e-aabb-94e3faac371e" (UID: "6e903702-6868-403e-aabb-94e3faac371e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:21:30.688856 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:30.688836    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e903702-6868-403e-aabb-94e3faac371e-home\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:21:30.688997 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:30.688983    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e903702-6868-403e-aabb-94e3faac371e-model-cache\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:21:30.689088 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:30.689077    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e903702-6868-403e-aabb-94e3faac371e-tmp-dir\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:21:30.691174 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:30.691127    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e903702-6868-403e-aabb-94e3faac371e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6e903702-6868-403e-aabb-94e3faac371e" (UID: "6e903702-6868-403e-aabb-94e3faac371e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:21:30.693930 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:30.692880    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e903702-6868-403e-aabb-94e3faac371e-dshm" (OuterVolumeSpecName: "dshm") pod "6e903702-6868-403e-aabb-94e3faac371e" (UID: "6e903702-6868-403e-aabb-94e3faac371e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:21:30.693930 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:30.693175    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e903702-6868-403e-aabb-94e3faac371e-kube-api-access-l4m9c" (OuterVolumeSpecName: "kube-api-access-l4m9c") pod "6e903702-6868-403e-aabb-94e3faac371e" (UID: "6e903702-6868-403e-aabb-94e3faac371e"). InnerVolumeSpecName "kube-api-access-l4m9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:21:30.790455 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:30.790431    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e903702-6868-403e-aabb-94e3faac371e-dshm\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:21:30.790455 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:30.790454    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e903702-6868-403e-aabb-94e3faac371e-tls-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:21:30.790455 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:30.790488    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4m9c\" (UniqueName: \"kubernetes.io/projected/6e903702-6868-403e-aabb-94e3faac371e-kube-api-access-l4m9c\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:21:30.974533 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:30.974495    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-079b38/e2e-conflict-a-9812c892-kserve-5655867767-qkchj"]
Sep 17 14:21:30.983191 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:30.983160    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-079b38/e2e-conflict-a-9812c892-kserve-5655867767-qkchj"]
Sep 17 14:21:31.571326 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:31.571289    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e903702-6868-403e-aabb-94e3faac371e" path="/var/lib/kubelet/pods/6e903702-6868-403e-aabb-94e3faac371e/volumes"
Sep 17 14:21:31.572716 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:31.572688    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:21:31.851918 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:31.851822    2560 scope.go:117] "RemoveContainer" containerID="3f4aab5fc7e782791aa422008c156fba1370a65755b90991183c40aea26a8738"
Sep 17 14:21:32.319346 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:32.319314    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:21:32.654115 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:32.654009    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:21:33.501444 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:33.501413    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:21:34.503959 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:34.503921    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:21:35.248242 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:35.248194    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:21:36.757543 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:36.757513    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-079b38/distinct-w4-079b38-kserve-7fd69fb686-5drgf"
Sep 17 14:21:36.757988 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:36.757574    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-079b38/distinct-w4-079b38-kserve-7fd69fb686-5drgf"
Sep 17 14:21:36.770337 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:36.770312    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-079b38/distinct-w4-079b38-kserve-7fd69fb686-5drgf"
Sep 17 14:21:37.292515 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:37.292481    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-5cdd59e7/e2e-lifecycle-model-5cdd59e7-kserve-655cf8b5dc-9rjlc"]
Sep 17 14:21:37.391031 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:37.390991    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:21:37.708695 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:37.708663    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-079b38/distinct-w4-079b38-kserve-7fd69fb686-5drgf"
Sep 17 14:21:52.311618 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:52.311563    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-5cdd59e7/e2e-lifecycle-model-5cdd59e7-kserve-655cf8b5dc-9rjlc" podUID="969c84d7-8c83-4483-ad5d-f52295b0dc2a" containerName="main" containerID="cri-o://06003c7d33d74ed284a3377baccd95e48376aab7c16a3e3a2f66fa55fda6c943" gracePeriod=45
Sep 17 14:21:52.569942 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:52.569891    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5cdd59e7/e2e-lifecycle-model-5cdd59e7-kserve-655cf8b5dc-9rjlc"
Sep 17 14:21:52.607887 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:52.607856    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/969c84d7-8c83-4483-ad5d-f52295b0dc2a-model-cache\") pod \"969c84d7-8c83-4483-ad5d-f52295b0dc2a\" (UID: \"969c84d7-8c83-4483-ad5d-f52295b0dc2a\") "
Sep 17 14:21:52.608048 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:52.608016    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cvtgp\" (UniqueName: \"kubernetes.io/projected/969c84d7-8c83-4483-ad5d-f52295b0dc2a-kube-api-access-cvtgp\") pod \"969c84d7-8c83-4483-ad5d-f52295b0dc2a\" (UID: \"969c84d7-8c83-4483-ad5d-f52295b0dc2a\") "
Sep 17 14:21:52.608108 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:52.608069    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/969c84d7-8c83-4483-ad5d-f52295b0dc2a-dshm\") pod \"969c84d7-8c83-4483-ad5d-f52295b0dc2a\" (UID: \"969c84d7-8c83-4483-ad5d-f52295b0dc2a\") "
Sep 17 14:21:52.608108 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:52.608094    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/969c84d7-8c83-4483-ad5d-f52295b0dc2a-tls-certs\") pod \"969c84d7-8c83-4483-ad5d-f52295b0dc2a\" (UID: \"969c84d7-8c83-4483-ad5d-f52295b0dc2a\") "
Sep 17 14:21:52.608108 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:52.608087    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/969c84d7-8c83-4483-ad5d-f52295b0dc2a-model-cache" (OuterVolumeSpecName: "model-cache") pod "969c84d7-8c83-4483-ad5d-f52295b0dc2a" (UID: "969c84d7-8c83-4483-ad5d-f52295b0dc2a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:21:52.608279 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:52.608140    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/969c84d7-8c83-4483-ad5d-f52295b0dc2a-home\") pod \"969c84d7-8c83-4483-ad5d-f52295b0dc2a\" (UID: \"969c84d7-8c83-4483-ad5d-f52295b0dc2a\") "
Sep 17 14:21:52.608279 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:52.608171    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/969c84d7-8c83-4483-ad5d-f52295b0dc2a-tmp-dir\") pod \"969c84d7-8c83-4483-ad5d-f52295b0dc2a\" (UID: \"969c84d7-8c83-4483-ad5d-f52295b0dc2a\") "
Sep 17 14:21:52.608605 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:52.608581    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/969c84d7-8c83-4483-ad5d-f52295b0dc2a-home" (OuterVolumeSpecName: "home") pod "969c84d7-8c83-4483-ad5d-f52295b0dc2a" (UID: "969c84d7-8c83-4483-ad5d-f52295b0dc2a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:21:52.608725 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:52.608608    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/969c84d7-8c83-4483-ad5d-f52295b0dc2a-model-cache\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:21:52.608790 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:52.608734    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/969c84d7-8c83-4483-ad5d-f52295b0dc2a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "969c84d7-8c83-4483-ad5d-f52295b0dc2a" (UID: "969c84d7-8c83-4483-ad5d-f52295b0dc2a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:21:52.610475 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:52.610415    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/969c84d7-8c83-4483-ad5d-f52295b0dc2a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "969c84d7-8c83-4483-ad5d-f52295b0dc2a" (UID: "969c84d7-8c83-4483-ad5d-f52295b0dc2a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:21:52.610765 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:52.610735    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/969c84d7-8c83-4483-ad5d-f52295b0dc2a-dshm" (OuterVolumeSpecName: "dshm") pod "969c84d7-8c83-4483-ad5d-f52295b0dc2a" (UID: "969c84d7-8c83-4483-ad5d-f52295b0dc2a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:21:52.610850 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:52.610792    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/969c84d7-8c83-4483-ad5d-f52295b0dc2a-kube-api-access-cvtgp" (OuterVolumeSpecName: "kube-api-access-cvtgp") pod "969c84d7-8c83-4483-ad5d-f52295b0dc2a" (UID: "969c84d7-8c83-4483-ad5d-f52295b0dc2a"). InnerVolumeSpecName "kube-api-access-cvtgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:21:52.709915 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:52.709881    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvtgp\" (UniqueName: \"kubernetes.io/projected/969c84d7-8c83-4483-ad5d-f52295b0dc2a-kube-api-access-cvtgp\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:21:52.709915 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:52.709910    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/969c84d7-8c83-4483-ad5d-f52295b0dc2a-dshm\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:21:52.709915 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:52.709920    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/969c84d7-8c83-4483-ad5d-f52295b0dc2a-tls-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:21:52.710146 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:52.709929    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/969c84d7-8c83-4483-ad5d-f52295b0dc2a-home\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:21:52.710146 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:52.709957    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/969c84d7-8c83-4483-ad5d-f52295b0dc2a-tmp-dir\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:21:52.795627 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:52.795594    2560 generic.go:356] "Generic (PLEG): container finished" podID="969c84d7-8c83-4483-ad5d-f52295b0dc2a" containerID="06003c7d33d74ed284a3377baccd95e48376aab7c16a3e3a2f66fa55fda6c943" exitCode=0
Sep 17 14:21:52.795781 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:52.795642    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5cdd59e7/e2e-lifecycle-model-5cdd59e7-kserve-655cf8b5dc-9rjlc"
Sep 17 14:21:52.795781 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:52.795639    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5cdd59e7/e2e-lifecycle-model-5cdd59e7-kserve-655cf8b5dc-9rjlc" event={"ID":"969c84d7-8c83-4483-ad5d-f52295b0dc2a","Type":"ContainerDied","Data":"06003c7d33d74ed284a3377baccd95e48376aab7c16a3e3a2f66fa55fda6c943"}
Sep 17 14:21:52.795910 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:52.795779    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5cdd59e7/e2e-lifecycle-model-5cdd59e7-kserve-655cf8b5dc-9rjlc" event={"ID":"969c84d7-8c83-4483-ad5d-f52295b0dc2a","Type":"ContainerDied","Data":"7e0a93d9f79c11a6fcda5dec8778b8a3ac72c833294275302644f7978525bc79"}
Sep 17 14:21:52.795910 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:52.795813    2560 scope.go:117] "RemoveContainer" containerID="06003c7d33d74ed284a3377baccd95e48376aab7c16a3e3a2f66fa55fda6c943"
Sep 17 14:21:52.805553 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:52.805532    2560 scope.go:117] "RemoveContainer" containerID="06003c7d33d74ed284a3377baccd95e48376aab7c16a3e3a2f66fa55fda6c943"
Sep 17 14:21:52.805818 ip-10-0-129-158 kubenswrapper[2560]: E0917 14:21:52.805799    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06003c7d33d74ed284a3377baccd95e48376aab7c16a3e3a2f66fa55fda6c943\": container with ID starting with 06003c7d33d74ed284a3377baccd95e48376aab7c16a3e3a2f66fa55fda6c943 not found: ID does not exist" containerID="06003c7d33d74ed284a3377baccd95e48376aab7c16a3e3a2f66fa55fda6c943"
Sep 17 14:21:52.805867 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:52.805825    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06003c7d33d74ed284a3377baccd95e48376aab7c16a3e3a2f66fa55fda6c943"} err="failed to get container status \"06003c7d33d74ed284a3377baccd95e48376aab7c16a3e3a2f66fa55fda6c943\": rpc error: code = NotFound desc = could not find container \"06003c7d33d74ed284a3377baccd95e48376aab7c16a3e3a2f66fa55fda6c943\": container with ID starting with 06003c7d33d74ed284a3377baccd95e48376aab7c16a3e3a2f66fa55fda6c943 not found: ID does not exist"
Sep 17 14:21:52.818911 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:52.818890    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-5cdd59e7/e2e-lifecycle-model-5cdd59e7-kserve-655cf8b5dc-9rjlc"]
Sep 17 14:21:52.823719 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:52.823670    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-5cdd59e7/e2e-lifecycle-model-5cdd59e7-kserve-655cf8b5dc-9rjlc"]
Sep 17 14:21:53.570605 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:53.570572    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="969c84d7-8c83-4483-ad5d-f52295b0dc2a" path="/var/lib/kubelet/pods/969c84d7-8c83-4483-ad5d-f52295b0dc2a/volumes"
Sep 17 14:21:56.699907 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:56.699872    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-b79988/auth-test-model-b79988-kserve-68d9ff4cbc-xlzsm"]
Sep 17 14:21:56.700357 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:56.700344    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e903702-6868-403e-aabb-94e3faac371e" containerName="main"
Sep 17 14:21:56.700357 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:56.700358    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e903702-6868-403e-aabb-94e3faac371e" containerName="main"
Sep 17 14:21:56.700448 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:56.700380    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="969c84d7-8c83-4483-ad5d-f52295b0dc2a" containerName="main"
Sep 17 14:21:56.700448 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:56.700386    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="969c84d7-8c83-4483-ad5d-f52295b0dc2a" containerName="main"
Sep 17 14:21:56.700550 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:56.700453    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="969c84d7-8c83-4483-ad5d-f52295b0dc2a" containerName="main"
Sep 17 14:21:56.700550 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:56.700476    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e903702-6868-403e-aabb-94e3faac371e" containerName="main"
Sep 17 14:21:56.704906 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:56.704890    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b79988/auth-test-model-b79988-kserve-68d9ff4cbc-xlzsm"
Sep 17 14:21:56.707766 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:56.707723    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-b79988\"/\"auth-test-model-b79988-kserve-self-signed-certs\""
Sep 17 14:21:56.707766 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:56.707744    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-b79988\"/\"default-dockercfg-gjn9m\""
Sep 17 14:21:56.707982 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:56.707803    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-b79988\"/\"openshift-service-ca.crt\""
Sep 17 14:21:56.707982 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:56.707849    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-b79988\"/\"kube-root-ca.crt\""
Sep 17 14:21:56.713771 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:56.713745    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b79988/auth-test-model-b79988-kserve-68d9ff4cbc-xlzsm"]
Sep 17 14:21:56.752298 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:56.752262    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b239de1-635b-4718-8349-6e5d5bf9f506-dshm\") pod \"auth-test-model-b79988-kserve-68d9ff4cbc-xlzsm\" (UID: \"9b239de1-635b-4718-8349-6e5d5bf9f506\") " pod="ai-tenant-e2e-shared-b-w3-b79988/auth-test-model-b79988-kserve-68d9ff4cbc-xlzsm"
Sep 17 14:21:56.752541 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:56.752312    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b239de1-635b-4718-8349-6e5d5bf9f506-model-cache\") pod \"auth-test-model-b79988-kserve-68d9ff4cbc-xlzsm\" (UID: \"9b239de1-635b-4718-8349-6e5d5bf9f506\") " pod="ai-tenant-e2e-shared-b-w3-b79988/auth-test-model-b79988-kserve-68d9ff4cbc-xlzsm"
Sep 17 14:21:56.752622 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:56.752533    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b239de1-635b-4718-8349-6e5d5bf9f506-tmp-dir\") pod \"auth-test-model-b79988-kserve-68d9ff4cbc-xlzsm\" (UID: \"9b239de1-635b-4718-8349-6e5d5bf9f506\") " pod="ai-tenant-e2e-shared-b-w3-b79988/auth-test-model-b79988-kserve-68d9ff4cbc-xlzsm"
Sep 17 14:21:56.752622 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:56.752584    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b239de1-635b-4718-8349-6e5d5bf9f506-tls-certs\") pod \"auth-test-model-b79988-kserve-68d9ff4cbc-xlzsm\" (UID: \"9b239de1-635b-4718-8349-6e5d5bf9f506\") " pod="ai-tenant-e2e-shared-b-w3-b79988/auth-test-model-b79988-kserve-68d9ff4cbc-xlzsm"
Sep 17 14:21:56.752733 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:56.752638    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ht4d\" (UniqueName: \"kubernetes.io/projected/9b239de1-635b-4718-8349-6e5d5bf9f506-kube-api-access-5ht4d\") pod \"auth-test-model-b79988-kserve-68d9ff4cbc-xlzsm\" (UID: \"9b239de1-635b-4718-8349-6e5d5bf9f506\") " pod="ai-tenant-e2e-shared-b-w3-b79988/auth-test-model-b79988-kserve-68d9ff4cbc-xlzsm"
Sep 17 14:21:56.752733 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:56.752668    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b239de1-635b-4718-8349-6e5d5bf9f506-home\") pod \"auth-test-model-b79988-kserve-68d9ff4cbc-xlzsm\" (UID: \"9b239de1-635b-4718-8349-6e5d5bf9f506\") " pod="ai-tenant-e2e-shared-b-w3-b79988/auth-test-model-b79988-kserve-68d9ff4cbc-xlzsm"
Sep 17 14:21:56.853325 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:56.853293    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ht4d\" (UniqueName: \"kubernetes.io/projected/9b239de1-635b-4718-8349-6e5d5bf9f506-kube-api-access-5ht4d\") pod \"auth-test-model-b79988-kserve-68d9ff4cbc-xlzsm\" (UID: \"9b239de1-635b-4718-8349-6e5d5bf9f506\") " pod="ai-tenant-e2e-shared-b-w3-b79988/auth-test-model-b79988-kserve-68d9ff4cbc-xlzsm"
Sep 17 14:21:56.853325 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:56.853328    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b239de1-635b-4718-8349-6e5d5bf9f506-home\") pod \"auth-test-model-b79988-kserve-68d9ff4cbc-xlzsm\" (UID: \"9b239de1-635b-4718-8349-6e5d5bf9f506\") " pod="ai-tenant-e2e-shared-b-w3-b79988/auth-test-model-b79988-kserve-68d9ff4cbc-xlzsm"
Sep 17 14:21:56.853607 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:56.853371    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b239de1-635b-4718-8349-6e5d5bf9f506-dshm\") pod \"auth-test-model-b79988-kserve-68d9ff4cbc-xlzsm\" (UID: \"9b239de1-635b-4718-8349-6e5d5bf9f506\") " pod="ai-tenant-e2e-shared-b-w3-b79988/auth-test-model-b79988-kserve-68d9ff4cbc-xlzsm"
Sep 17 14:21:56.853607 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:56.853394    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b239de1-635b-4718-8349-6e5d5bf9f506-model-cache\") pod \"auth-test-model-b79988-kserve-68d9ff4cbc-xlzsm\" (UID: \"9b239de1-635b-4718-8349-6e5d5bf9f506\") " pod="ai-tenant-e2e-shared-b-w3-b79988/auth-test-model-b79988-kserve-68d9ff4cbc-xlzsm"
Sep 17 14:21:56.853607 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:56.853542    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b239de1-635b-4718-8349-6e5d5bf9f506-tmp-dir\") pod \"auth-test-model-b79988-kserve-68d9ff4cbc-xlzsm\" (UID: \"9b239de1-635b-4718-8349-6e5d5bf9f506\") " pod="ai-tenant-e2e-shared-b-w3-b79988/auth-test-model-b79988-kserve-68d9ff4cbc-xlzsm"
Sep 17 14:21:56.853607 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:56.853578    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b239de1-635b-4718-8349-6e5d5bf9f506-tls-certs\") pod \"auth-test-model-b79988-kserve-68d9ff4cbc-xlzsm\" (UID: \"9b239de1-635b-4718-8349-6e5d5bf9f506\") " pod="ai-tenant-e2e-shared-b-w3-b79988/auth-test-model-b79988-kserve-68d9ff4cbc-xlzsm"
Sep 17 14:21:56.853817 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:56.853762    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b239de1-635b-4718-8349-6e5d5bf9f506-home\") pod \"auth-test-model-b79988-kserve-68d9ff4cbc-xlzsm\" (UID: \"9b239de1-635b-4718-8349-6e5d5bf9f506\") " pod="ai-tenant-e2e-shared-b-w3-b79988/auth-test-model-b79988-kserve-68d9ff4cbc-xlzsm"
Sep 17 14:21:56.853931 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:56.853914    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b239de1-635b-4718-8349-6e5d5bf9f506-tmp-dir\") pod \"auth-test-model-b79988-kserve-68d9ff4cbc-xlzsm\" (UID: \"9b239de1-635b-4718-8349-6e5d5bf9f506\") " pod="ai-tenant-e2e-shared-b-w3-b79988/auth-test-model-b79988-kserve-68d9ff4cbc-xlzsm"
Sep 17 14:21:56.853990 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:56.853914    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b239de1-635b-4718-8349-6e5d5bf9f506-model-cache\") pod \"auth-test-model-b79988-kserve-68d9ff4cbc-xlzsm\" (UID: \"9b239de1-635b-4718-8349-6e5d5bf9f506\") " pod="ai-tenant-e2e-shared-b-w3-b79988/auth-test-model-b79988-kserve-68d9ff4cbc-xlzsm"
Sep 17 14:21:56.855730 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:56.855709    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b239de1-635b-4718-8349-6e5d5bf9f506-dshm\") pod \"auth-test-model-b79988-kserve-68d9ff4cbc-xlzsm\" (UID: \"9b239de1-635b-4718-8349-6e5d5bf9f506\") " pod="ai-tenant-e2e-shared-b-w3-b79988/auth-test-model-b79988-kserve-68d9ff4cbc-xlzsm"
Sep 17 14:21:56.855853 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:56.855837    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b239de1-635b-4718-8349-6e5d5bf9f506-tls-certs\") pod \"auth-test-model-b79988-kserve-68d9ff4cbc-xlzsm\" (UID: \"9b239de1-635b-4718-8349-6e5d5bf9f506\") " pod="ai-tenant-e2e-shared-b-w3-b79988/auth-test-model-b79988-kserve-68d9ff4cbc-xlzsm"
Sep 17 14:21:56.861512 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:56.861486    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ht4d\" (UniqueName: \"kubernetes.io/projected/9b239de1-635b-4718-8349-6e5d5bf9f506-kube-api-access-5ht4d\") pod \"auth-test-model-b79988-kserve-68d9ff4cbc-xlzsm\" (UID: \"9b239de1-635b-4718-8349-6e5d5bf9f506\") " pod="ai-tenant-e2e-shared-b-w3-b79988/auth-test-model-b79988-kserve-68d9ff4cbc-xlzsm"
Sep 17 14:21:57.009944 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:57.009857    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:21:57.015085 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:57.015060    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b79988/auth-test-model-b79988-kserve-68d9ff4cbc-xlzsm"
Sep 17 14:21:57.318016 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:57.317988    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b79988/auth-test-model-b79988-kserve-68d9ff4cbc-xlzsm"]
Sep 17 14:21:57.321024 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:21:57.320997    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9b239de1_635b_4718_8349_6e5d5bf9f506.slice/crio-7b5051dff7ba6304a317c9eeaa168b728f7482e35ccd562ae3b0218eda701404 WatchSource:0}: Error finding container 7b5051dff7ba6304a317c9eeaa168b728f7482e35ccd562ae3b0218eda701404: Status 404 returned error can't find the container with id 7b5051dff7ba6304a317c9eeaa168b728f7482e35ccd562ae3b0218eda701404
Sep 17 14:21:57.833995 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:57.833959    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b79988/auth-test-model-b79988-kserve-68d9ff4cbc-xlzsm" event={"ID":"9b239de1-635b-4718-8349-6e5d5bf9f506","Type":"ContainerStarted","Data":"1518bbb163fbe167a109a4328ed1bd0f8a4fa33d236d59ef74c068890890b7b1"}
Sep 17 14:21:57.834362 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:57.834003    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b79988/auth-test-model-b79988-kserve-68d9ff4cbc-xlzsm" event={"ID":"9b239de1-635b-4718-8349-6e5d5bf9f506","Type":"ContainerStarted","Data":"7b5051dff7ba6304a317c9eeaa168b728f7482e35ccd562ae3b0218eda701404"}
Sep 17 14:21:57.858028 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:21:57.857979    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-b79988/auth-test-model-b79988-kserve-68d9ff4cbc-xlzsm" podStartSLOduration=1.85796424 podStartE2EDuration="1.85796424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:21:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:21:57.854659439 +0000 UTC m=+1586.796803085" watchObservedRunningTime="2026-09-17 14:21:57.85796424 +0000 UTC m=+1586.800107949"
Sep 17 14:22:00.604068 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:00.604034    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:02.276056 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:02.276015    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:07.015961 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:07.015929    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-b79988/auth-test-model-b79988-kserve-68d9ff4cbc-xlzsm"
Sep 17 14:22:07.016577 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:07.015979    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-b79988/auth-test-model-b79988-kserve-68d9ff4cbc-xlzsm"
Sep 17 14:22:07.028845 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:07.028823    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-b79988/auth-test-model-b79988-kserve-68d9ff4cbc-xlzsm"
Sep 17 14:22:07.909433 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:07.909404    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-b79988/auth-test-model-b79988-kserve-68d9ff4cbc-xlzsm"
Sep 17 14:22:13.739718 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:13.739665    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:15.372475 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:15.372430    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:21.509406 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:21.509372    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:37.129224 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:37.129188    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:37.381346 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:37.381268    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h"]
Sep 17 14:22:37.381545 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:37.381504    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h" podUID="98a693c5-386f-4525-b3e1-7f891ef7d68a" containerName="istio-proxy" containerID="cri-o://ac64c1cf1c6dfcae08f3df2a809ef845eb6d9c8f176d92d7f1add7f46cdcbbeb" gracePeriod=30
Sep 17 14:22:38.310895 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:38.310859    2560 patch_prober.go:28] interesting pod/e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 17 14:22:38.311324 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:38.310922    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h" podUID="98a693c5-386f-4525-b3e1-7f891ef7d68a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 14:22:38.599451 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:38.599367    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:38.708182 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:38.708142    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:38.792605 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:38.792573    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:39.482634 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:39.482596    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:39.579519 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:39.579445    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:39.679599 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:39.679567    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:40.939599 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:40.939540    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:42.644697 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:42.644635    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h"
Sep 17 14:22:42.807230 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:42.807198    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98a693c5-386f-4525-b3e1-7f891ef7d68a-istio-envoy\") pod \"98a693c5-386f-4525-b3e1-7f891ef7d68a\" (UID: \"98a693c5-386f-4525-b3e1-7f891ef7d68a\") "
Sep 17 14:22:42.807230 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:42.807232    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98a693c5-386f-4525-b3e1-7f891ef7d68a-workload-certs\") pod \"98a693c5-386f-4525-b3e1-7f891ef7d68a\" (UID: \"98a693c5-386f-4525-b3e1-7f891ef7d68a\") "
Sep 17 14:22:42.807476 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:42.807260    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pvqbr\" (UniqueName: \"kubernetes.io/projected/98a693c5-386f-4525-b3e1-7f891ef7d68a-kube-api-access-pvqbr\") pod \"98a693c5-386f-4525-b3e1-7f891ef7d68a\" (UID: \"98a693c5-386f-4525-b3e1-7f891ef7d68a\") "
Sep 17 14:22:42.807476 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:42.807313    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98a693c5-386f-4525-b3e1-7f891ef7d68a-istio-podinfo\") pod \"98a693c5-386f-4525-b3e1-7f891ef7d68a\" (UID: \"98a693c5-386f-4525-b3e1-7f891ef7d68a\") "
Sep 17 14:22:42.807476 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:42.807340    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98a693c5-386f-4525-b3e1-7f891ef7d68a-istio-data\") pod \"98a693c5-386f-4525-b3e1-7f891ef7d68a\" (UID: \"98a693c5-386f-4525-b3e1-7f891ef7d68a\") "
Sep 17 14:22:42.807476 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:42.807417    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98a693c5-386f-4525-b3e1-7f891ef7d68a-istiod-ca-cert\") pod \"98a693c5-386f-4525-b3e1-7f891ef7d68a\" (UID: \"98a693c5-386f-4525-b3e1-7f891ef7d68a\") "
Sep 17 14:22:42.807712 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:42.807517    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98a693c5-386f-4525-b3e1-7f891ef7d68a-credential-socket\") pod \"98a693c5-386f-4525-b3e1-7f891ef7d68a\" (UID: \"98a693c5-386f-4525-b3e1-7f891ef7d68a\") "
Sep 17 14:22:42.807712 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:42.807519    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98a693c5-386f-4525-b3e1-7f891ef7d68a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "98a693c5-386f-4525-b3e1-7f891ef7d68a" (UID: "98a693c5-386f-4525-b3e1-7f891ef7d68a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:22:42.807712 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:42.807579    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98a693c5-386f-4525-b3e1-7f891ef7d68a-workload-socket\") pod \"98a693c5-386f-4525-b3e1-7f891ef7d68a\" (UID: \"98a693c5-386f-4525-b3e1-7f891ef7d68a\") "
Sep 17 14:22:42.807712 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:42.807609    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98a693c5-386f-4525-b3e1-7f891ef7d68a-istio-token\") pod \"98a693c5-386f-4525-b3e1-7f891ef7d68a\" (UID: \"98a693c5-386f-4525-b3e1-7f891ef7d68a\") "
Sep 17 14:22:42.807984 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:42.807722    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98a693c5-386f-4525-b3e1-7f891ef7d68a-istio-data" (OuterVolumeSpecName: "istio-data") pod "98a693c5-386f-4525-b3e1-7f891ef7d68a" (UID: "98a693c5-386f-4525-b3e1-7f891ef7d68a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:22:42.807984 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:42.807822    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98a693c5-386f-4525-b3e1-7f891ef7d68a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "98a693c5-386f-4525-b3e1-7f891ef7d68a" (UID: "98a693c5-386f-4525-b3e1-7f891ef7d68a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:22:42.807984 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:42.807869    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98a693c5-386f-4525-b3e1-7f891ef7d68a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "98a693c5-386f-4525-b3e1-7f891ef7d68a" (UID: "98a693c5-386f-4525-b3e1-7f891ef7d68a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:22:42.807984 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:42.807929    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98a693c5-386f-4525-b3e1-7f891ef7d68a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "98a693c5-386f-4525-b3e1-7f891ef7d68a" (UID: "98a693c5-386f-4525-b3e1-7f891ef7d68a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:22:42.808205 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:42.808087    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98a693c5-386f-4525-b3e1-7f891ef7d68a-credential-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:22:42.808205 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:42.808101    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98a693c5-386f-4525-b3e1-7f891ef7d68a-workload-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:22:42.808205 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:42.808111    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98a693c5-386f-4525-b3e1-7f891ef7d68a-workload-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:22:42.808205 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:42.808119    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98a693c5-386f-4525-b3e1-7f891ef7d68a-istio-data\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:22:42.808205 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:42.808132    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98a693c5-386f-4525-b3e1-7f891ef7d68a-istiod-ca-cert\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:22:42.809681 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:42.809648    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98a693c5-386f-4525-b3e1-7f891ef7d68a-kube-api-access-pvqbr" (OuterVolumeSpecName: "kube-api-access-pvqbr") pod "98a693c5-386f-4525-b3e1-7f891ef7d68a" (UID: "98a693c5-386f-4525-b3e1-7f891ef7d68a"). InnerVolumeSpecName "kube-api-access-pvqbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:22:42.809793 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:42.809706    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98a693c5-386f-4525-b3e1-7f891ef7d68a-istio-token" (OuterVolumeSpecName: "istio-token") pod "98a693c5-386f-4525-b3e1-7f891ef7d68a" (UID: "98a693c5-386f-4525-b3e1-7f891ef7d68a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:22:42.809887 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:42.809870    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98a693c5-386f-4525-b3e1-7f891ef7d68a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "98a693c5-386f-4525-b3e1-7f891ef7d68a" (UID: "98a693c5-386f-4525-b3e1-7f891ef7d68a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:22:42.809938 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:42.809885    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/98a693c5-386f-4525-b3e1-7f891ef7d68a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "98a693c5-386f-4525-b3e1-7f891ef7d68a" (UID: "98a693c5-386f-4525-b3e1-7f891ef7d68a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 14:22:42.908998 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:42.908933    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98a693c5-386f-4525-b3e1-7f891ef7d68a-istio-token\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:22:42.908998 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:42.908955    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98a693c5-386f-4525-b3e1-7f891ef7d68a-istio-envoy\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:22:42.908998 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:42.908964    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvqbr\" (UniqueName: \"kubernetes.io/projected/98a693c5-386f-4525-b3e1-7f891ef7d68a-kube-api-access-pvqbr\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:22:42.908998 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:42.908974    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98a693c5-386f-4525-b3e1-7f891ef7d68a-istio-podinfo\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:22:43.130635 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:43.130599    2560 generic.go:356] "Generic (PLEG): container finished" podID="98a693c5-386f-4525-b3e1-7f891ef7d68a" containerID="ac64c1cf1c6dfcae08f3df2a809ef845eb6d9c8f176d92d7f1add7f46cdcbbeb" exitCode=0
Sep 17 14:22:43.130831 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:43.130655    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h"
Sep 17 14:22:43.130831 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:43.130674    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h" event={"ID":"98a693c5-386f-4525-b3e1-7f891ef7d68a","Type":"ContainerDied","Data":"ac64c1cf1c6dfcae08f3df2a809ef845eb6d9c8f176d92d7f1add7f46cdcbbeb"}
Sep 17 14:22:43.130831 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:43.130751    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h" event={"ID":"98a693c5-386f-4525-b3e1-7f891ef7d68a","Type":"ContainerDied","Data":"37f877121aa145ac3f5fcf8f8aad779e2ec4c37713d8470519f22d4b4b042b08"}
Sep 17 14:22:43.130831 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:43.130775    2560 scope.go:117] "RemoveContainer" containerID="ac64c1cf1c6dfcae08f3df2a809ef845eb6d9c8f176d92d7f1add7f46cdcbbeb"
Sep 17 14:22:43.141192 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:43.141176    2560 scope.go:117] "RemoveContainer" containerID="6cb7b2daac5fd38d7fa3dda92140b61a7e71ce7da14c7accd5cab2dd3bd4b32b"
Sep 17 14:22:43.150284 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:43.150266    2560 scope.go:117] "RemoveContainer" containerID="ac64c1cf1c6dfcae08f3df2a809ef845eb6d9c8f176d92d7f1add7f46cdcbbeb"
Sep 17 14:22:43.150541 ip-10-0-129-158 kubenswrapper[2560]: E0917 14:22:43.150520    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac64c1cf1c6dfcae08f3df2a809ef845eb6d9c8f176d92d7f1add7f46cdcbbeb\": container with ID starting with ac64c1cf1c6dfcae08f3df2a809ef845eb6d9c8f176d92d7f1add7f46cdcbbeb not found: ID does not exist" containerID="ac64c1cf1c6dfcae08f3df2a809ef845eb6d9c8f176d92d7f1add7f46cdcbbeb"
Sep 17 14:22:43.150589 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:43.150548    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac64c1cf1c6dfcae08f3df2a809ef845eb6d9c8f176d92d7f1add7f46cdcbbeb"} err="failed to get container status \"ac64c1cf1c6dfcae08f3df2a809ef845eb6d9c8f176d92d7f1add7f46cdcbbeb\": rpc error: code = NotFound desc = could not find container \"ac64c1cf1c6dfcae08f3df2a809ef845eb6d9c8f176d92d7f1add7f46cdcbbeb\": container with ID starting with ac64c1cf1c6dfcae08f3df2a809ef845eb6d9c8f176d92d7f1add7f46cdcbbeb not found: ID does not exist"
Sep 17 14:22:43.150589 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:43.150568    2560 scope.go:117] "RemoveContainer" containerID="6cb7b2daac5fd38d7fa3dda92140b61a7e71ce7da14c7accd5cab2dd3bd4b32b"
Sep 17 14:22:43.150769 ip-10-0-129-158 kubenswrapper[2560]: E0917 14:22:43.150754    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cb7b2daac5fd38d7fa3dda92140b61a7e71ce7da14c7accd5cab2dd3bd4b32b\": container with ID starting with 6cb7b2daac5fd38d7fa3dda92140b61a7e71ce7da14c7accd5cab2dd3bd4b32b not found: ID does not exist" containerID="6cb7b2daac5fd38d7fa3dda92140b61a7e71ce7da14c7accd5cab2dd3bd4b32b"
Sep 17 14:22:43.150814 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:43.150772    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cb7b2daac5fd38d7fa3dda92140b61a7e71ce7da14c7accd5cab2dd3bd4b32b"} err="failed to get container status \"6cb7b2daac5fd38d7fa3dda92140b61a7e71ce7da14c7accd5cab2dd3bd4b32b\": rpc error: code = NotFound desc = could not find container \"6cb7b2daac5fd38d7fa3dda92140b61a7e71ce7da14c7accd5cab2dd3bd4b32b\": container with ID starting with 6cb7b2daac5fd38d7fa3dda92140b61a7e71ce7da14c7accd5cab2dd3bd4b32b not found: ID does not exist"
Sep 17 14:22:43.155931 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:43.155905    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h"]
Sep 17 14:22:43.159396 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:43.159346    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5cdd59e7-openshift-default-79794f765b-v742h"]
Sep 17 14:22:43.569764 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:43.569730    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98a693c5-386f-4525-b3e1-7f891ef7d68a" path="/var/lib/kubelet/pods/98a693c5-386f-4525-b3e1-7f891ef7d68a/volumes"
Sep 17 14:22:44.429781 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:44.429750    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-079b38/distinct-2-w4-079b38-kserve-5546bf445c-kv4gf"]
Sep 17 14:22:44.430314 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:44.430296    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98a693c5-386f-4525-b3e1-7f891ef7d68a" containerName="istio-proxy"
Sep 17 14:22:44.430379 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:44.430317    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="98a693c5-386f-4525-b3e1-7f891ef7d68a" containerName="istio-proxy"
Sep 17 14:22:44.430433 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:44.430423    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="98a693c5-386f-4525-b3e1-7f891ef7d68a" containerName="istio-proxy"
Sep 17 14:22:44.430484 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:44.430436    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="98a693c5-386f-4525-b3e1-7f891ef7d68a" containerName="istio-proxy"
Sep 17 14:22:44.435189 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:44.435168    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-079b38/distinct-2-w4-079b38-kserve-5546bf445c-kv4gf"
Sep 17 14:22:44.437445 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:44.437426    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-079b38\"/\"distinct-2-w4-079b38-kserve-self-signed-certs\""
Sep 17 14:22:44.444492 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:44.444450    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-079b38/distinct-2-w4-079b38-kserve-5546bf445c-kv4gf"]
Sep 17 14:22:44.459945 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:44.459922    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:44.527917 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:44.527893    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bfb040f-2e96-4649-a866-73f7c960c31d-model-cache\") pod \"distinct-2-w4-079b38-kserve-5546bf445c-kv4gf\" (UID: \"8bfb040f-2e96-4649-a866-73f7c960c31d\") " pod="e2e-models-e2e-worker-w4-079b38/distinct-2-w4-079b38-kserve-5546bf445c-kv4gf"
Sep 17 14:22:44.528033 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:44.527928    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h9b9\" (UniqueName: \"kubernetes.io/projected/8bfb040f-2e96-4649-a866-73f7c960c31d-kube-api-access-9h9b9\") pod \"distinct-2-w4-079b38-kserve-5546bf445c-kv4gf\" (UID: \"8bfb040f-2e96-4649-a866-73f7c960c31d\") " pod="e2e-models-e2e-worker-w4-079b38/distinct-2-w4-079b38-kserve-5546bf445c-kv4gf"
Sep 17 14:22:44.528033 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:44.527971    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bfb040f-2e96-4649-a866-73f7c960c31d-dshm\") pod \"distinct-2-w4-079b38-kserve-5546bf445c-kv4gf\" (UID: \"8bfb040f-2e96-4649-a866-73f7c960c31d\") " pod="e2e-models-e2e-worker-w4-079b38/distinct-2-w4-079b38-kserve-5546bf445c-kv4gf"
Sep 17 14:22:44.528033 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:44.527998    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bfb040f-2e96-4649-a866-73f7c960c31d-tls-certs\") pod \"distinct-2-w4-079b38-kserve-5546bf445c-kv4gf\" (UID: \"8bfb040f-2e96-4649-a866-73f7c960c31d\") " pod="e2e-models-e2e-worker-w4-079b38/distinct-2-w4-079b38-kserve-5546bf445c-kv4gf"
Sep 17 14:22:44.528183 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:44.528060    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bfb040f-2e96-4649-a866-73f7c960c31d-home\") pod \"distinct-2-w4-079b38-kserve-5546bf445c-kv4gf\" (UID: \"8bfb040f-2e96-4649-a866-73f7c960c31d\") " pod="e2e-models-e2e-worker-w4-079b38/distinct-2-w4-079b38-kserve-5546bf445c-kv4gf"
Sep 17 14:22:44.528183 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:44.528097    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bfb040f-2e96-4649-a866-73f7c960c31d-tmp-dir\") pod \"distinct-2-w4-079b38-kserve-5546bf445c-kv4gf\" (UID: \"8bfb040f-2e96-4649-a866-73f7c960c31d\") " pod="e2e-models-e2e-worker-w4-079b38/distinct-2-w4-079b38-kserve-5546bf445c-kv4gf"
Sep 17 14:22:44.628858 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:44.628824    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bfb040f-2e96-4649-a866-73f7c960c31d-home\") pod \"distinct-2-w4-079b38-kserve-5546bf445c-kv4gf\" (UID: \"8bfb040f-2e96-4649-a866-73f7c960c31d\") " pod="e2e-models-e2e-worker-w4-079b38/distinct-2-w4-079b38-kserve-5546bf445c-kv4gf"
Sep 17 14:22:44.629029 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:44.628880    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bfb040f-2e96-4649-a866-73f7c960c31d-tmp-dir\") pod \"distinct-2-w4-079b38-kserve-5546bf445c-kv4gf\" (UID: \"8bfb040f-2e96-4649-a866-73f7c960c31d\") " pod="e2e-models-e2e-worker-w4-079b38/distinct-2-w4-079b38-kserve-5546bf445c-kv4gf"
Sep 17 14:22:44.629029 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:44.628939    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bfb040f-2e96-4649-a866-73f7c960c31d-model-cache\") pod \"distinct-2-w4-079b38-kserve-5546bf445c-kv4gf\" (UID: \"8bfb040f-2e96-4649-a866-73f7c960c31d\") " pod="e2e-models-e2e-worker-w4-079b38/distinct-2-w4-079b38-kserve-5546bf445c-kv4gf"
Sep 17 14:22:44.629029 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:44.628962    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h9b9\" (UniqueName: \"kubernetes.io/projected/8bfb040f-2e96-4649-a866-73f7c960c31d-kube-api-access-9h9b9\") pod \"distinct-2-w4-079b38-kserve-5546bf445c-kv4gf\" (UID: \"8bfb040f-2e96-4649-a866-73f7c960c31d\") " pod="e2e-models-e2e-worker-w4-079b38/distinct-2-w4-079b38-kserve-5546bf445c-kv4gf"
Sep 17 14:22:44.629029 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:44.629002    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bfb040f-2e96-4649-a866-73f7c960c31d-dshm\") pod \"distinct-2-w4-079b38-kserve-5546bf445c-kv4gf\" (UID: \"8bfb040f-2e96-4649-a866-73f7c960c31d\") " pod="e2e-models-e2e-worker-w4-079b38/distinct-2-w4-079b38-kserve-5546bf445c-kv4gf"
Sep 17 14:22:44.629029 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:44.629022    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bfb040f-2e96-4649-a866-73f7c960c31d-tls-certs\") pod \"distinct-2-w4-079b38-kserve-5546bf445c-kv4gf\" (UID: \"8bfb040f-2e96-4649-a866-73f7c960c31d\") " pod="e2e-models-e2e-worker-w4-079b38/distinct-2-w4-079b38-kserve-5546bf445c-kv4gf"
Sep 17 14:22:44.629258 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:44.629205    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bfb040f-2e96-4649-a866-73f7c960c31d-home\") pod \"distinct-2-w4-079b38-kserve-5546bf445c-kv4gf\" (UID: \"8bfb040f-2e96-4649-a866-73f7c960c31d\") " pod="e2e-models-e2e-worker-w4-079b38/distinct-2-w4-079b38-kserve-5546bf445c-kv4gf"
Sep 17 14:22:44.629318 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:44.629286    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bfb040f-2e96-4649-a866-73f7c960c31d-model-cache\") pod \"distinct-2-w4-079b38-kserve-5546bf445c-kv4gf\" (UID: \"8bfb040f-2e96-4649-a866-73f7c960c31d\") " pod="e2e-models-e2e-worker-w4-079b38/distinct-2-w4-079b38-kserve-5546bf445c-kv4gf"
Sep 17 14:22:44.629374 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:44.629350    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bfb040f-2e96-4649-a866-73f7c960c31d-tmp-dir\") pod \"distinct-2-w4-079b38-kserve-5546bf445c-kv4gf\" (UID: \"8bfb040f-2e96-4649-a866-73f7c960c31d\") " pod="e2e-models-e2e-worker-w4-079b38/distinct-2-w4-079b38-kserve-5546bf445c-kv4gf"
Sep 17 14:22:44.631255 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:44.631237    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bfb040f-2e96-4649-a866-73f7c960c31d-dshm\") pod \"distinct-2-w4-079b38-kserve-5546bf445c-kv4gf\" (UID: \"8bfb040f-2e96-4649-a866-73f7c960c31d\") " pod="e2e-models-e2e-worker-w4-079b38/distinct-2-w4-079b38-kserve-5546bf445c-kv4gf"
Sep 17 14:22:44.631611 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:44.631596    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bfb040f-2e96-4649-a866-73f7c960c31d-tls-certs\") pod \"distinct-2-w4-079b38-kserve-5546bf445c-kv4gf\" (UID: \"8bfb040f-2e96-4649-a866-73f7c960c31d\") " pod="e2e-models-e2e-worker-w4-079b38/distinct-2-w4-079b38-kserve-5546bf445c-kv4gf"
Sep 17 14:22:44.637992 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:44.637973    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h9b9\" (UniqueName: \"kubernetes.io/projected/8bfb040f-2e96-4649-a866-73f7c960c31d-kube-api-access-9h9b9\") pod \"distinct-2-w4-079b38-kserve-5546bf445c-kv4gf\" (UID: \"8bfb040f-2e96-4649-a866-73f7c960c31d\") " pod="e2e-models-e2e-worker-w4-079b38/distinct-2-w4-079b38-kserve-5546bf445c-kv4gf"
Sep 17 14:22:44.746495 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:44.746375    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-079b38/distinct-2-w4-079b38-kserve-5546bf445c-kv4gf"
Sep 17 14:22:45.034187 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:45.034160    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-079b38/distinct-2-w4-079b38-kserve-5546bf445c-kv4gf"]
Sep 17 14:22:45.035983 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:22:45.035954    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8bfb040f_2e96_4649_a866_73f7c960c31d.slice/crio-ba0748217f15e4b5abd4475d91c5f8dc3d9a165130d564530fc7900ee8444db6 WatchSource:0}: Error finding container ba0748217f15e4b5abd4475d91c5f8dc3d9a165130d564530fc7900ee8444db6: Status 404 returned error can't find the container with id ba0748217f15e4b5abd4475d91c5f8dc3d9a165130d564530fc7900ee8444db6
Sep 17 14:22:45.145531 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:45.145495    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-079b38/distinct-2-w4-079b38-kserve-5546bf445c-kv4gf" event={"ID":"8bfb040f-2e96-4649-a866-73f7c960c31d","Type":"ContainerStarted","Data":"9beb2b8e48da358394aae4f2a8dcd56422c7680e26b345642b4efca85e121626"}
Sep 17 14:22:45.145678 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:45.145547    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-079b38/distinct-2-w4-079b38-kserve-5546bf445c-kv4gf" event={"ID":"8bfb040f-2e96-4649-a866-73f7c960c31d","Type":"ContainerStarted","Data":"ba0748217f15e4b5abd4475d91c5f8dc3d9a165130d564530fc7900ee8444db6"}
Sep 17 14:22:45.167918 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:45.167874    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-079b38/distinct-2-w4-079b38-kserve-5546bf445c-kv4gf" podStartSLOduration=1.167860394 podStartE2EDuration="1.167860394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:22:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:22:45.165648219 +0000 UTC m=+1634.107791866" watchObservedRunningTime="2026-09-17 14:22:45.167860394 +0000 UTC m=+1634.110004073"
Sep 17 14:22:47.302198 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:47.302146    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:47.371351 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:47.371316    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:47.993166 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:47.993140    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:50.016825 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:50.016787    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:54.747385 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:54.747348    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-079b38/distinct-2-w4-079b38-kserve-5546bf445c-kv4gf"
Sep 17 14:22:54.747818 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:54.747403    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-079b38/distinct-2-w4-079b38-kserve-5546bf445c-kv4gf"
Sep 17 14:22:54.760213 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:54.760184    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-079b38/distinct-2-w4-079b38-kserve-5546bf445c-kv4gf"
Sep 17 14:22:55.220475 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:55.220439    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-079b38/distinct-2-w4-079b38-kserve-5546bf445c-kv4gf"
Sep 17 14:22:55.536788 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:55.536709    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:55.637992 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:55.637956    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:55.739040 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:55.739003    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:58.735544 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:22:58.735505    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:23:06.200244 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:23:06.200197    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:23:06.421826 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:23:06.421787    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:23:07.098177 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:23:07.098141    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-d9b17d/embedding-w0-d9b17d-kserve-547f78f6c5-vhlvm"]
Sep 17 14:23:07.098621 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:23:07.098603    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98a693c5-386f-4525-b3e1-7f891ef7d68a" containerName="istio-proxy"
Sep 17 14:23:07.098621 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:23:07.098619    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="98a693c5-386f-4525-b3e1-7f891ef7d68a" containerName="istio-proxy"
Sep 17 14:23:07.101853 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:23:07.101836    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d9b17d/embedding-w0-d9b17d-kserve-547f78f6c5-vhlvm"
Sep 17 14:23:07.104040 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:23:07.104023    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d9b17d\"/\"embedding-w0-d9b17d-kserve-self-signed-certs\""
Sep 17 14:23:07.111410 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:23:07.111390    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d9b17d/embedding-w0-d9b17d-kserve-547f78f6c5-vhlvm"]
Sep 17 14:23:07.129427 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:23:07.129400    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:23:07.146707 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:23:07.146680    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/668b3d6d-a30f-4949-bd22-292b896116f6-tmp-dir\") pod \"embedding-w0-d9b17d-kserve-547f78f6c5-vhlvm\" (UID: \"668b3d6d-a30f-4949-bd22-292b896116f6\") " pod="e2e-models-e2e-worker-w0-d9b17d/embedding-w0-d9b17d-kserve-547f78f6c5-vhlvm"
Sep 17 14:23:07.146843 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:23:07.146766    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/668b3d6d-a30f-4949-bd22-292b896116f6-tls-certs\") pod \"embedding-w0-d9b17d-kserve-547f78f6c5-vhlvm\" (UID: \"668b3d6d-a30f-4949-bd22-292b896116f6\") " pod="e2e-models-e2e-worker-w0-d9b17d/embedding-w0-d9b17d-kserve-547f78f6c5-vhlvm"
Sep 17 14:23:07.146843 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:23:07.146802    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/668b3d6d-a30f-4949-bd22-292b896116f6-dshm\") pod \"embedding-w0-d9b17d-kserve-547f78f6c5-vhlvm\" (UID: \"668b3d6d-a30f-4949-bd22-292b896116f6\") " pod="e2e-models-e2e-worker-w0-d9b17d/embedding-w0-d9b17d-kserve-547f78f6c5-vhlvm"
Sep 17 14:23:07.146961 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:23:07.146847    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/668b3d6d-a30f-4949-bd22-292b896116f6-home\") pod \"embedding-w0-d9b17d-kserve-547f78f6c5-vhlvm\" (UID: \"668b3d6d-a30f-4949-bd22-292b896116f6\") " pod="e2e-models-e2e-worker-w0-d9b17d/embedding-w0-d9b17d-kserve-547f78f6c5-vhlvm"
Sep 17 14:23:07.146961 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:23:07.146876    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdgqd\" (UniqueName: \"kubernetes.io/projected/668b3d6d-a30f-4949-bd22-292b896116f6-kube-api-access-tdgqd\") pod \"embedding-w0-d9b17d-kserve-547f78f6c5-vhlvm\" (UID: \"668b3d6d-a30f-4949-bd22-292b896116f6\") " pod="e2e-models-e2e-worker-w0-d9b17d/embedding-w0-d9b17d-kserve-547f78f6c5-vhlvm"
Sep 17 14:23:07.146961 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:23:07.146932    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/668b3d6d-a30f-4949-bd22-292b896116f6-model-cache\") pod \"embedding-w0-d9b17d-kserve-547f78f6c5-vhlvm\" (UID: \"668b3d6d-a30f-4949-bd22-292b896116f6\") " pod="e2e-models-e2e-worker-w0-d9b17d/embedding-w0-d9b17d-kserve-547f78f6c5-vhlvm"
Sep 17 14:23:07.248177 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:23:07.248154    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/668b3d6d-a30f-4949-bd22-292b896116f6-tmp-dir\") pod \"embedding-w0-d9b17d-kserve-547f78f6c5-vhlvm\" (UID: \"668b3d6d-a30f-4949-bd22-292b896116f6\") " pod="e2e-models-e2e-worker-w0-d9b17d/embedding-w0-d9b17d-kserve-547f78f6c5-vhlvm"
Sep 17 14:23:07.248588 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:23:07.248203    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/668b3d6d-a30f-4949-bd22-292b896116f6-tls-certs\") pod \"embedding-w0-d9b17d-kserve-547f78f6c5-vhlvm\" (UID: \"668b3d6d-a30f-4949-bd22-292b896116f6\") " pod="e2e-models-e2e-worker-w0-d9b17d/embedding-w0-d9b17d-kserve-547f78f6c5-vhlvm"
Sep 17 14:23:07.248588 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:23:07.248226    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/668b3d6d-a30f-4949-bd22-292b896116f6-dshm\") pod \"embedding-w0-d9b17d-kserve-547f78f6c5-vhlvm\" (UID: \"668b3d6d-a30f-4949-bd22-292b896116f6\") " pod="e2e-models-e2e-worker-w0-d9b17d/embedding-w0-d9b17d-kserve-547f78f6c5-vhlvm"
Sep 17 14:23:07.248588 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:23:07.248264    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/668b3d6d-a30f-4949-bd22-292b896116f6-home\") pod \"embedding-w0-d9b17d-kserve-547f78f6c5-vhlvm\" (UID: \"668b3d6d-a30f-4949-bd22-292b896116f6\") " pod="e2e-models-e2e-worker-w0-d9b17d/embedding-w0-d9b17d-kserve-547f78f6c5-vhlvm"
Sep 17 14:23:07.248588 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:23:07.248283    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdgqd\" (UniqueName: \"kubernetes.io/projected/668b3d6d-a30f-4949-bd22-292b896116f6-kube-api-access-tdgqd\") pod \"embedding-w0-d9b17d-kserve-547f78f6c5-vhlvm\" (UID: \"668b3d6d-a30f-4949-bd22-292b896116f6\") " pod="e2e-models-e2e-worker-w0-d9b17d/embedding-w0-d9b17d-kserve-547f78f6c5-vhlvm"
Sep 17 14:23:07.248588 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:23:07.248317    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/668b3d6d-a30f-4949-bd22-292b896116f6-model-cache\") pod \"embedding-w0-d9b17d-kserve-547f78f6c5-vhlvm\" (UID: \"668b3d6d-a30f-4949-bd22-292b896116f6\") " pod="e2e-models-e2e-worker-w0-d9b17d/embedding-w0-d9b17d-kserve-547f78f6c5-vhlvm"
Sep 17 14:23:07.248865 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:23:07.248621    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/668b3d6d-a30f-4949-bd22-292b896116f6-tmp-dir\") pod \"embedding-w0-d9b17d-kserve-547f78f6c5-vhlvm\" (UID: \"668b3d6d-a30f-4949-bd22-292b896116f6\") " pod="e2e-models-e2e-worker-w0-d9b17d/embedding-w0-d9b17d-kserve-547f78f6c5-vhlvm"
Sep 17 14:23:07.248865 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:23:07.248717    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/668b3d6d-a30f-4949-bd22-292b896116f6-home\") pod \"embedding-w0-d9b17d-kserve-547f78f6c5-vhlvm\" (UID: \"668b3d6d-a30f-4949-bd22-292b896116f6\") " pod="e2e-models-e2e-worker-w0-d9b17d/embedding-w0-d9b17d-kserve-547f78f6c5-vhlvm"
Sep 17 14:23:07.248865 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:23:07.248779    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/668b3d6d-a30f-4949-bd22-292b896116f6-model-cache\") pod \"embedding-w0-d9b17d-kserve-547f78f6c5-vhlvm\" (UID: \"668b3d6d-a30f-4949-bd22-292b896116f6\") " pod="e2e-models-e2e-worker-w0-d9b17d/embedding-w0-d9b17d-kserve-547f78f6c5-vhlvm"
Sep 17 14:23:07.250529 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:23:07.250506    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/668b3d6d-a30f-4949-bd22-292b896116f6-dshm\") pod \"embedding-w0-d9b17d-kserve-547f78f6c5-vhlvm\" (UID: \"668b3d6d-a30f-4949-bd22-292b896116f6\") " pod="e2e-models-e2e-worker-w0-d9b17d/embedding-w0-d9b17d-kserve-547f78f6c5-vhlvm"
Sep 17 14:23:07.250807 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:23:07.250789    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/668b3d6d-a30f-4949-bd22-292b896116f6-tls-certs\") pod \"embedding-w0-d9b17d-kserve-547f78f6c5-vhlvm\" (UID: \"668b3d6d-a30f-4949-bd22-292b896116f6\") " pod="e2e-models-e2e-worker-w0-d9b17d/embedding-w0-d9b17d-kserve-547f78f6c5-vhlvm"
Sep 17 14:23:07.255680 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:23:07.255660    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdgqd\" (UniqueName: \"kubernetes.io/projected/668b3d6d-a30f-4949-bd22-292b896116f6-kube-api-access-tdgqd\") pod \"embedding-w0-d9b17d-kserve-547f78f6c5-vhlvm\" (UID: \"668b3d6d-a30f-4949-bd22-292b896116f6\") " pod="e2e-models-e2e-worker-w0-d9b17d/embedding-w0-d9b17d-kserve-547f78f6c5-vhlvm"
Sep 17 14:23:07.412293 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:23:07.412211    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d9b17d/embedding-w0-d9b17d-kserve-547f78f6c5-vhlvm"
Sep 17 14:23:07.702296 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:23:07.702260    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d9b17d/embedding-w0-d9b17d-kserve-547f78f6c5-vhlvm"]
Sep 17 14:23:07.703132 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:23:07.703108    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod668b3d6d_a30f_4949_bd22_292b896116f6.slice/crio-276b70d62c97af83dc310588729186ba7bc0b725ec839272b8455ccbe9d37c75 WatchSource:0}: Error finding container 276b70d62c97af83dc310588729186ba7bc0b725ec839272b8455ccbe9d37c75: Status 404 returned error can't find the container with id 276b70d62c97af83dc310588729186ba7bc0b725ec839272b8455ccbe9d37c75
Sep 17 14:23:08.297676 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:23:08.297639    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d9b17d/embedding-w0-d9b17d-kserve-547f78f6c5-vhlvm" event={"ID":"668b3d6d-a30f-4949-bd22-292b896116f6","Type":"ContainerStarted","Data":"52b556f56edea34deb819d5707c0436afd6943db7b5184fef77be6b37fc944b6"}
Sep 17 14:23:08.297676 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:23:08.297680    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d9b17d/embedding-w0-d9b17d-kserve-547f78f6c5-vhlvm" event={"ID":"668b3d6d-a30f-4949-bd22-292b896116f6","Type":"ContainerStarted","Data":"276b70d62c97af83dc310588729186ba7bc0b725ec839272b8455ccbe9d37c75"}
Sep 17 14:23:08.318708 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:23:08.318655    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-d9b17d/embedding-w0-d9b17d-kserve-547f78f6c5-vhlvm" podStartSLOduration=1.318640613 podStartE2EDuration="1.318640613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:23:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:23:08.316481744 +0000 UTC m=+1657.258625386" watchObservedRunningTime="2026-09-17 14:23:08.318640613 +0000 UTC m=+1657.260784259"
Sep 17 14:23:17.412553 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:23:17.412519    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-d9b17d/embedding-w0-d9b17d-kserve-547f78f6c5-vhlvm"
Sep 17 14:23:17.413092 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:23:17.412569    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d9b17d/embedding-w0-d9b17d-kserve-547f78f6c5-vhlvm"
Sep 17 14:23:17.424588 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:23:17.424562    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-d9b17d/embedding-w0-d9b17d-kserve-547f78f6c5-vhlvm"
Sep 17 14:23:18.375589 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:23:18.375560    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d9b17d/embedding-w0-d9b17d-kserve-547f78f6c5-vhlvm"
Sep 17 14:23:24.057322 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:23:24.057264    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:23:27.343856 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:23:27.343817    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:23:31.904613 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:23:31.904575    2560 scope.go:117] "RemoveContainer" containerID="47d1d9fd19cc65bfc5a1a1f6b4c6489b8ca534819be28ad3a6abb1db01586d60"
Sep 17 14:23:31.914012 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:23:31.913995    2560 scope.go:117] "RemoveContainer" containerID="87d7fc26f3de138e9850b59c88efac3684b674305f87a492199005aed3b9f8ac"
Sep 17 14:23:34.290613 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:23:34.290573    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:23:34.351657 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:23:34.351624    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:23:34.636203 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:23:34.636151    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:23:34.922521 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:23:34.922430    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:23:35.318320 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:23:35.318285    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:23:37.456140 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:23:37.456106    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:23:38.060165 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:23:38.060127    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:23:38.148356 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:23:38.148303    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:23:38.405549 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:23:38.405432    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:23:39.045439 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:23:39.045402    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:23:39.315997 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:23:39.315964    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:23:40.140347 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:23:40.140312    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:23:40.942204 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:23:40.942164    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:23:41.743512 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:23:41.743470    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:23:42.678611 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:23:42.678576    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:23:43.665612 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:23:43.665579    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:23:43.770656 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:23:43.770618    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:23:43.898873 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:23:43.898835    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:23:44.230663 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:23:44.230632    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:23:44.335402 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:23:44.335357    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:05.137263 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:05.137230    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:06.146125 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:06.146095    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:07.998519 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:07.998484    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:08.127294 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:08.127248    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:08.593864 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:08.593827    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:08.936756 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:08.936682    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv"]
Sep 17 14:24:08.940381 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:08.940365    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv"
Sep 17 14:24:08.943695 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:08.943674    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-a77c87-openshift-default-dockercfg-rlfxn\""
Sep 17 14:24:08.954881 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:08.954855    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv"]
Sep 17 14:24:08.957944 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:08.957923    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:08.990385 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:08.990361    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f909bf57-b7b6-4e2b-b054-e06b7385a90a-workload-certs\") pod \"e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv\" (UID: \"f909bf57-b7b6-4e2b-b054-e06b7385a90a\") " pod="openshift-ingress/e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv"
Sep 17 14:24:08.990525 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:08.990396    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d26w\" (UniqueName: \"kubernetes.io/projected/f909bf57-b7b6-4e2b-b054-e06b7385a90a-kube-api-access-8d26w\") pod \"e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv\" (UID: \"f909bf57-b7b6-4e2b-b054-e06b7385a90a\") " pod="openshift-ingress/e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv"
Sep 17 14:24:08.990525 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:08.990442    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f909bf57-b7b6-4e2b-b054-e06b7385a90a-istio-envoy\") pod \"e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv\" (UID: \"f909bf57-b7b6-4e2b-b054-e06b7385a90a\") " pod="openshift-ingress/e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv"
Sep 17 14:24:08.990525 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:08.990497    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f909bf57-b7b6-4e2b-b054-e06b7385a90a-istio-token\") pod \"e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv\" (UID: \"f909bf57-b7b6-4e2b-b054-e06b7385a90a\") " pod="openshift-ingress/e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv"
Sep 17 14:24:08.990632 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:08.990541    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f909bf57-b7b6-4e2b-b054-e06b7385a90a-credential-socket\") pod \"e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv\" (UID: \"f909bf57-b7b6-4e2b-b054-e06b7385a90a\") " pod="openshift-ingress/e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv"
Sep 17 14:24:08.990632 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:08.990568    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f909bf57-b7b6-4e2b-b054-e06b7385a90a-istio-data\") pod \"e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv\" (UID: \"f909bf57-b7b6-4e2b-b054-e06b7385a90a\") " pod="openshift-ingress/e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv"
Sep 17 14:24:08.990632 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:08.990585    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f909bf57-b7b6-4e2b-b054-e06b7385a90a-istio-podinfo\") pod \"e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv\" (UID: \"f909bf57-b7b6-4e2b-b054-e06b7385a90a\") " pod="openshift-ingress/e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv"
Sep 17 14:24:08.990632 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:08.990615    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f909bf57-b7b6-4e2b-b054-e06b7385a90a-istiod-ca-cert\") pod \"e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv\" (UID: \"f909bf57-b7b6-4e2b-b054-e06b7385a90a\") " pod="openshift-ingress/e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv"
Sep 17 14:24:08.990763 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:08.990668    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f909bf57-b7b6-4e2b-b054-e06b7385a90a-workload-socket\") pod \"e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv\" (UID: \"f909bf57-b7b6-4e2b-b054-e06b7385a90a\") " pod="openshift-ingress/e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv"
Sep 17 14:24:09.057075 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:09.057047    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:09.091888 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:09.091864    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f909bf57-b7b6-4e2b-b054-e06b7385a90a-workload-certs\") pod \"e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv\" (UID: \"f909bf57-b7b6-4e2b-b054-e06b7385a90a\") " pod="openshift-ingress/e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv"
Sep 17 14:24:09.092016 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:09.091905    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d26w\" (UniqueName: \"kubernetes.io/projected/f909bf57-b7b6-4e2b-b054-e06b7385a90a-kube-api-access-8d26w\") pod \"e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv\" (UID: \"f909bf57-b7b6-4e2b-b054-e06b7385a90a\") " pod="openshift-ingress/e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv"
Sep 17 14:24:09.092016 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:09.091979    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f909bf57-b7b6-4e2b-b054-e06b7385a90a-istio-envoy\") pod \"e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv\" (UID: \"f909bf57-b7b6-4e2b-b054-e06b7385a90a\") " pod="openshift-ingress/e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv"
Sep 17 14:24:09.092132 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:09.092015    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f909bf57-b7b6-4e2b-b054-e06b7385a90a-istio-token\") pod \"e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv\" (UID: \"f909bf57-b7b6-4e2b-b054-e06b7385a90a\") " pod="openshift-ingress/e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv"
Sep 17 14:24:09.092132 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:09.092050    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f909bf57-b7b6-4e2b-b054-e06b7385a90a-credential-socket\") pod \"e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv\" (UID: \"f909bf57-b7b6-4e2b-b054-e06b7385a90a\") " pod="openshift-ingress/e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv"
Sep 17 14:24:09.092132 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:09.092075    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f909bf57-b7b6-4e2b-b054-e06b7385a90a-istio-data\") pod \"e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv\" (UID: \"f909bf57-b7b6-4e2b-b054-e06b7385a90a\") " pod="openshift-ingress/e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv"
Sep 17 14:24:09.092132 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:09.092100    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f909bf57-b7b6-4e2b-b054-e06b7385a90a-istio-podinfo\") pod \"e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv\" (UID: \"f909bf57-b7b6-4e2b-b054-e06b7385a90a\") " pod="openshift-ingress/e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv"
Sep 17 14:24:09.092355 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:09.092145    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f909bf57-b7b6-4e2b-b054-e06b7385a90a-istiod-ca-cert\") pod \"e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv\" (UID: \"f909bf57-b7b6-4e2b-b054-e06b7385a90a\") " pod="openshift-ingress/e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv"
Sep 17 14:24:09.092355 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:09.092210    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f909bf57-b7b6-4e2b-b054-e06b7385a90a-workload-socket\") pod \"e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv\" (UID: \"f909bf57-b7b6-4e2b-b054-e06b7385a90a\") " pod="openshift-ingress/e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv"
Sep 17 14:24:09.092355 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:09.092249    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f909bf57-b7b6-4e2b-b054-e06b7385a90a-workload-certs\") pod \"e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv\" (UID: \"f909bf57-b7b6-4e2b-b054-e06b7385a90a\") " pod="openshift-ingress/e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv"
Sep 17 14:24:09.092562 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:09.092420    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f909bf57-b7b6-4e2b-b054-e06b7385a90a-credential-socket\") pod \"e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv\" (UID: \"f909bf57-b7b6-4e2b-b054-e06b7385a90a\") " pod="openshift-ingress/e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv"
Sep 17 14:24:09.092562 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:09.092521    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f909bf57-b7b6-4e2b-b054-e06b7385a90a-istio-data\") pod \"e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv\" (UID: \"f909bf57-b7b6-4e2b-b054-e06b7385a90a\") " pod="openshift-ingress/e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv"
Sep 17 14:24:09.092670 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:09.092608    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f909bf57-b7b6-4e2b-b054-e06b7385a90a-workload-socket\") pod \"e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv\" (UID: \"f909bf57-b7b6-4e2b-b054-e06b7385a90a\") " pod="openshift-ingress/e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv"
Sep 17 14:24:09.092909 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:09.092891    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f909bf57-b7b6-4e2b-b054-e06b7385a90a-istiod-ca-cert\") pod \"e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv\" (UID: \"f909bf57-b7b6-4e2b-b054-e06b7385a90a\") " pod="openshift-ingress/e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv"
Sep 17 14:24:09.094340 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:09.094315    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f909bf57-b7b6-4e2b-b054-e06b7385a90a-istio-envoy\") pod \"e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv\" (UID: \"f909bf57-b7b6-4e2b-b054-e06b7385a90a\") " pod="openshift-ingress/e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv"
Sep 17 14:24:09.094521 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:09.094506    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f909bf57-b7b6-4e2b-b054-e06b7385a90a-istio-podinfo\") pod \"e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv\" (UID: \"f909bf57-b7b6-4e2b-b054-e06b7385a90a\") " pod="openshift-ingress/e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv"
Sep 17 14:24:09.100503 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:09.100479    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d26w\" (UniqueName: \"kubernetes.io/projected/f909bf57-b7b6-4e2b-b054-e06b7385a90a-kube-api-access-8d26w\") pod \"e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv\" (UID: \"f909bf57-b7b6-4e2b-b054-e06b7385a90a\") " pod="openshift-ingress/e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv"
Sep 17 14:24:09.100586 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:09.100540    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f909bf57-b7b6-4e2b-b054-e06b7385a90a-istio-token\") pod \"e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv\" (UID: \"f909bf57-b7b6-4e2b-b054-e06b7385a90a\") " pod="openshift-ingress/e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv"
Sep 17 14:24:09.251490 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:09.251375    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv"
Sep 17 14:24:09.575072 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:09.575045    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv"]
Sep 17 14:24:09.576369 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:24:09.576347    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf909bf57_b7b6_4e2b_b054_e06b7385a90a.slice/crio-732c2996668dc9579885d0258ae4247e1f1096b8bb262fb60cdaf5a050e964fc WatchSource:0}: Error finding container 732c2996668dc9579885d0258ae4247e1f1096b8bb262fb60cdaf5a050e964fc: Status 404 returned error can't find the container with id 732c2996668dc9579885d0258ae4247e1f1096b8bb262fb60cdaf5a050e964fc
Sep 17 14:24:09.579044 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:09.579013    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:24:09.579147 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:09.579086    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:24:09.579147 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:09.579131    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:24:09.721404 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:09.721365    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv" event={"ID":"f909bf57-b7b6-4e2b-b054-e06b7385a90a","Type":"ContainerStarted","Data":"1575f29d6dd2f965f8c0aa0c391d03ea8d9cb24fa1ce73e6ef735a21ceb58cf4"}
Sep 17 14:24:09.721562 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:09.721426    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv" event={"ID":"f909bf57-b7b6-4e2b-b054-e06b7385a90a","Type":"ContainerStarted","Data":"732c2996668dc9579885d0258ae4247e1f1096b8bb262fb60cdaf5a050e964fc"}
Sep 17 14:24:09.739289 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:09.739253    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:09.753745 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:09.753679    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv" podStartSLOduration=1.7536638 podStartE2EDuration="1.7536638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:24:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:24:09.752105915 +0000 UTC m=+1718.694249564" watchObservedRunningTime="2026-09-17 14:24:09.7536638 +0000 UTC m=+1718.695807445"
Sep 17 14:24:10.063436 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:10.063402    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:10.252402 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:10.252373    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv"
Sep 17 14:24:10.257145 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:10.257120    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv"
Sep 17 14:24:10.729311 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:10.729283    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv"
Sep 17 14:24:10.730233 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:10.730215    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv"
Sep 17 14:24:11.282819 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:11.282790    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:11.733823 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:11.733783    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:11.841627 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:11.841583    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:12.324676 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:12.324638    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:13.357248 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:13.357210    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:14.163060 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:14.163021    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:15.029760 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:15.029725    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:15.980441 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:15.980408    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:16.281088 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:16.281002    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:16.379409 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:16.379369    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:16.596615 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:16.596566    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:16.691937 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:16.691906    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:17.052683 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:17.052649    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:17.940450 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:17.940418    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:18.843803 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:18.843765    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:19.580817 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:19.580775    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:31.944521 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:31.944492    2560 scope.go:117] "RemoveContainer" containerID="e475bad21a307163f58afbc78700721c1e8c68e471c606b0218df85bdd1ee0ad"
Sep 17 14:24:38.643126 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:38.643091    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:39.440623 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:39.440581    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:39.850166 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:39.850109    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:39.951866 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:39.951838    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d9b17d-5j4h4"]
Sep 17 14:24:39.955655 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:39.955637    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d9b17d-5j4h4"
Sep 17 14:24:39.957973 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:39.957955    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rp9ww\""
Sep 17 14:24:39.958074 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:39.957959    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 14:24:39.963743 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:39.963718    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d9b17d-5j4h4"]
Sep 17 14:24:40.047048 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:40.047009    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af34693e-c05d-4894-96cb-19a6523871a1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-d9b17d-5j4h4\" (UID: \"af34693e-c05d-4894-96cb-19a6523871a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d9b17d-5j4h4"
Sep 17 14:24:40.148401 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:40.148302    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af34693e-c05d-4894-96cb-19a6523871a1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-d9b17d-5j4h4\" (UID: \"af34693e-c05d-4894-96cb-19a6523871a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d9b17d-5j4h4"
Sep 17 14:24:40.148943 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:40.148921    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af34693e-c05d-4894-96cb-19a6523871a1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-d9b17d-5j4h4\" (UID: \"af34693e-c05d-4894-96cb-19a6523871a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d9b17d-5j4h4"
Sep 17 14:24:40.269774 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:40.269731    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d9b17d-5j4h4"
Sep 17 14:24:40.520253 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:40.520156    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:40.549191 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:40.549161    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d9b17d-5j4h4"]
Sep 17 14:24:40.549982 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:24:40.549944    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf34693e_c05d_4894_96cb_19a6523871a1.slice/crio-36edbd74e3f5afee75696d5c1a1a6412212d3d8e548132c19e036c62a4e3ff7b WatchSource:0}: Error finding container 36edbd74e3f5afee75696d5c1a1a6412212d3d8e548132c19e036c62a4e3ff7b: Status 404 returned error can't find the container with id 36edbd74e3f5afee75696d5c1a1a6412212d3d8e548132c19e036c62a4e3ff7b
Sep 17 14:24:40.938199 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:40.938161    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d9b17d-5j4h4" event={"ID":"af34693e-c05d-4894-96cb-19a6523871a1","Type":"ContainerStarted","Data":"960f6baef91ecb59711855aee9510d37eaacde69d6f60d379dd7bf00bf87d9d5"}
Sep 17 14:24:40.938573 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:40.938227    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d9b17d-5j4h4" event={"ID":"af34693e-c05d-4894-96cb-19a6523871a1","Type":"ContainerStarted","Data":"36edbd74e3f5afee75696d5c1a1a6412212d3d8e548132c19e036c62a4e3ff7b"}
Sep 17 14:24:40.955824 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:40.955762    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d9b17d-5j4h4" podStartSLOduration=1.9557453470000001 podStartE2EDuration="1.955745347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:24:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:24:40.955112856 +0000 UTC m=+1749.897256504" watchObservedRunningTime="2026-09-17 14:24:40.955745347 +0000 UTC m=+1749.897888994"
Sep 17 14:24:41.522226 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:41.522136    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:41.770608 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:41.770568    2560 patch_prober.go:28] interesting pod/e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 17 14:24:41.770775 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:41.770621    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv" podUID="f909bf57-b7b6-4e2b-b054-e06b7385a90a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 17 14:24:41.798670 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:41.798597    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:41.946745 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:41.946715    2560 generic.go:356] "Generic (PLEG): container finished" podID="af34693e-c05d-4894-96cb-19a6523871a1" containerID="960f6baef91ecb59711855aee9510d37eaacde69d6f60d379dd7bf00bf87d9d5" exitCode=0
Sep 17 14:24:41.947116 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:41.946753    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d9b17d-5j4h4" event={"ID":"af34693e-c05d-4894-96cb-19a6523871a1","Type":"ContainerDied","Data":"960f6baef91ecb59711855aee9510d37eaacde69d6f60d379dd7bf00bf87d9d5"}
Sep 17 14:24:42.423358 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:42.423324    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:42.954623 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:42.954592    2560 generic.go:356] "Generic (PLEG): container finished" podID="f909bf57-b7b6-4e2b-b054-e06b7385a90a" containerID="1575f29d6dd2f965f8c0aa0c391d03ea8d9cb24fa1ce73e6ef735a21ceb58cf4" exitCode=0
Sep 17 14:24:42.955110 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:42.954661    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv" event={"ID":"f909bf57-b7b6-4e2b-b054-e06b7385a90a","Type":"ContainerDied","Data":"1575f29d6dd2f965f8c0aa0c391d03ea8d9cb24fa1ce73e6ef735a21ceb58cf4"}
Sep 17 14:24:42.955435 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:42.955173    2560 scope.go:117] "RemoveContainer" containerID="1575f29d6dd2f965f8c0aa0c391d03ea8d9cb24fa1ce73e6ef735a21ceb58cf4"
Sep 17 14:24:42.956374 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:42.956349    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:24:42.956486 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:42.956419    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:24:42.956486 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:42.956451    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:24:43.111448 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:43.111421    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d9b17d-5j4h4"
Sep 17 14:24:43.182456 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:43.182425    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af34693e-c05d-4894-96cb-19a6523871a1-maas-api-service-ca\") pod \"af34693e-c05d-4894-96cb-19a6523871a1\" (UID: \"af34693e-c05d-4894-96cb-19a6523871a1\") "
Sep 17 14:24:43.182764 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:43.182741    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af34693e-c05d-4894-96cb-19a6523871a1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "af34693e-c05d-4894-96cb-19a6523871a1" (UID: "af34693e-c05d-4894-96cb-19a6523871a1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:24:43.183019 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:43.182998    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af34693e-c05d-4894-96cb-19a6523871a1-maas-api-service-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:24:43.325971 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:43.325936    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:43.967740 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:43.967619    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d9b17d-5j4h4" event={"ID":"af34693e-c05d-4894-96cb-19a6523871a1","Type":"ContainerDied","Data":"36edbd74e3f5afee75696d5c1a1a6412212d3d8e548132c19e036c62a4e3ff7b"}
Sep 17 14:24:43.968322 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:43.968283    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36edbd74e3f5afee75696d5c1a1a6412212d3d8e548132c19e036c62a4e3ff7b"
Sep 17 14:24:43.968585 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:43.968567    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d9b17d-5j4h4"
Sep 17 14:24:43.976480 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:43.973007    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv" event={"ID":"f909bf57-b7b6-4e2b-b054-e06b7385a90a","Type":"ContainerStarted","Data":"50ea7864495aa34217865296bc4db68b5473ed5b503c5c346021f98ffc032e35"}
Sep 17 14:24:44.183570 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:44.183528    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:44.252029 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:44.251952    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv"
Sep 17 14:24:44.252188 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:44.252129    2560 patch_prober.go:28] interesting pod/e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Sep 17 14:24:44.252238 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:44.252195    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv" podUID="f909bf57-b7b6-4e2b-b054-e06b7385a90a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Sep 17 14:24:44.630712 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:44.630679    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6fce83-2tt5z"]
Sep 17 14:24:44.631187 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:44.631172    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af34693e-c05d-4894-96cb-19a6523871a1" containerName="revoke-keys"
Sep 17 14:24:44.631240 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:44.631189    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="af34693e-c05d-4894-96cb-19a6523871a1" containerName="revoke-keys"
Sep 17 14:24:44.631296 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:44.631287    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="af34693e-c05d-4894-96cb-19a6523871a1" containerName="revoke-keys"
Sep 17 14:24:44.634560 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:44.634537    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6fce83-2tt5z"
Sep 17 14:24:44.636797 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:44.636776    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 14:24:44.636885 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:44.636779    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rp9ww\""
Sep 17 14:24:44.642056 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:44.642033    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6fce83-2tt5z"]
Sep 17 14:24:44.801435 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:44.801401    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b625e8de-13a9-4b72-b87f-079af3240401-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6fce83-2tt5z\" (UID: \"b625e8de-13a9-4b72-b87f-079af3240401\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6fce83-2tt5z"
Sep 17 14:24:44.815024 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:44.814996    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d9b17d/distinct-2-w0-d9b17d-kserve-56497687d9-7x45h"]
Sep 17 14:24:44.820173 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:44.820141    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d9b17d/distinct-w0-d9b17d-kserve-85bc8c598b-cflt6"]
Sep 17 14:24:44.825654 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:44.825628    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d9b17d/embedding-w0-d9b17d-kserve-547f78f6c5-vhlvm"]
Sep 17 14:24:44.833258 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:44.833226    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d9b17d/premium-w0-d9b17d-kserve-54647796d7-4r49f"]
Sep 17 14:24:44.903212 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:44.903130    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b625e8de-13a9-4b72-b87f-079af3240401-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6fce83-2tt5z\" (UID: \"b625e8de-13a9-4b72-b87f-079af3240401\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6fce83-2tt5z"
Sep 17 14:24:44.903772 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:44.903755    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b625e8de-13a9-4b72-b87f-079af3240401-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6fce83-2tt5z\" (UID: \"b625e8de-13a9-4b72-b87f-079af3240401\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6fce83-2tt5z"
Sep 17 14:24:44.946158 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:44.946133    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6fce83-2tt5z"
Sep 17 14:24:45.036804 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:45.036766    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:45.177655 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:45.177514    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:45.245602 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:45.245575    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6fce83-2tt5z"]
Sep 17 14:24:45.245720 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:24:45.245646    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb625e8de_13a9_4b72_b87f_079af3240401.slice/crio-10244805e20bef72c05c43da10b9ca181d34a87f535b5d4f6d7fcfd2bac83928 WatchSource:0}: Error finding container 10244805e20bef72c05c43da10b9ca181d34a87f535b5d4f6d7fcfd2bac83928: Status 404 returned error can't find the container with id 10244805e20bef72c05c43da10b9ca181d34a87f535b5d4f6d7fcfd2bac83928
Sep 17 14:24:45.269514 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:45.262005    2560 patch_prober.go:28] interesting pod/e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Sep 17 14:24:45.269514 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:45.262069    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv" podUID="f909bf57-b7b6-4e2b-b054-e06b7385a90a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Sep 17 14:24:45.999207 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:45.999131    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6fce83-2tt5z" event={"ID":"b625e8de-13a9-4b72-b87f-079af3240401","Type":"ContainerStarted","Data":"849cadc71389223ce66a35f570aa8a1e1179652601076f0159c043596fa8276b"}
Sep 17 14:24:45.999207 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:45.999187    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6fce83-2tt5z" event={"ID":"b625e8de-13a9-4b72-b87f-079af3240401","Type":"ContainerStarted","Data":"10244805e20bef72c05c43da10b9ca181d34a87f535b5d4f6d7fcfd2bac83928"}
Sep 17 14:24:46.015354 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:46.015307    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6fce83-2tt5z" podStartSLOduration=2.01529374 podStartE2EDuration="2.01529374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:24:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:24:46.01329109 +0000 UTC m=+1754.955434735" watchObservedRunningTime="2026-09-17 14:24:46.01529374 +0000 UTC m=+1754.957437386"
Sep 17 14:24:46.050018 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:46.049990    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:46.251941 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:46.251856    2560 patch_prober.go:28] interesting pod/e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Sep 17 14:24:46.251941 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:46.251910    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv" podUID="f909bf57-b7b6-4e2b-b054-e06b7385a90a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Sep 17 14:24:46.927892 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:46.927854    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:47.007695 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:47.007660    2560 generic.go:356] "Generic (PLEG): container finished" podID="b625e8de-13a9-4b72-b87f-079af3240401" containerID="849cadc71389223ce66a35f570aa8a1e1179652601076f0159c043596fa8276b" exitCode=0
Sep 17 14:24:47.007920 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:47.007745    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6fce83-2tt5z" event={"ID":"b625e8de-13a9-4b72-b87f-079af3240401","Type":"ContainerDied","Data":"849cadc71389223ce66a35f570aa8a1e1179652601076f0159c043596fa8276b"}
Sep 17 14:24:47.251871 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:47.251791    2560 patch_prober.go:28] interesting pod/e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Sep 17 14:24:47.251871 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:47.251842    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv" podUID="f909bf57-b7b6-4e2b-b054-e06b7385a90a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Sep 17 14:24:47.782360 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:47.782300    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:48.186669 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:48.186639    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6fce83-2tt5z"
Sep 17 14:24:48.251850 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:48.251816    2560 patch_prober.go:28] interesting pod/e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Sep 17 14:24:48.251998 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:48.251874    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv" podUID="f909bf57-b7b6-4e2b-b054-e06b7385a90a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Sep 17 14:24:48.344190 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:48.344117    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b625e8de-13a9-4b72-b87f-079af3240401-maas-api-service-ca\") pod \"b625e8de-13a9-4b72-b87f-079af3240401\" (UID: \"b625e8de-13a9-4b72-b87f-079af3240401\") "
Sep 17 14:24:48.344437 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:48.344415    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b625e8de-13a9-4b72-b87f-079af3240401-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b625e8de-13a9-4b72-b87f-079af3240401" (UID: "b625e8de-13a9-4b72-b87f-079af3240401"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:24:48.445069 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:48.445039    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b625e8de-13a9-4b72-b87f-079af3240401-maas-api-service-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:24:48.622745 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:48.622634    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:49.021914 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:49.021838    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6fce83-2tt5z"
Sep 17 14:24:49.021914 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:49.021883    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6fce83-2tt5z" event={"ID":"b625e8de-13a9-4b72-b87f-079af3240401","Type":"ContainerDied","Data":"10244805e20bef72c05c43da10b9ca181d34a87f535b5d4f6d7fcfd2bac83928"}
Sep 17 14:24:49.022091 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:49.021929    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10244805e20bef72c05c43da10b9ca181d34a87f535b5d4f6d7fcfd2bac83928"
Sep 17 14:24:49.252436 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:49.252398    2560 patch_prober.go:28] interesting pod/e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Sep 17 14:24:49.252900 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:49.252456    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv" podUID="f909bf57-b7b6-4e2b-b054-e06b7385a90a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Sep 17 14:24:49.341055 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:49.341023    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:50.215188 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:50.215145    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:50.255428 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:50.255403    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv"
Sep 17 14:24:50.255895 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:50.255814    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv"
Sep 17 14:24:50.256528 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:50.256509    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv"
Sep 17 14:24:50.975311 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:50.975280    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:51.746961 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:51.746931    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:52.544006 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:52.543961    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:53.413736 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:53.413702    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:54.779272 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:54.779233    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:58.214768 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:58.214732    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:58.220640 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:58.220614    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d9b17d-29827575-44kg8"]
Sep 17 14:24:58.223564 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:58.223540    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d9b17d-29827575-44kg8"]
Sep 17 14:24:58.258253 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:58.258222    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-d9b17d-67659db75d-fl9f9"]
Sep 17 14:24:58.258500 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:58.258453    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d9b17d-67659db75d-fl9f9" podUID="87759bfd-b7d8-45e0-b5f0-8d5c081abaf3" containerName="payload-pre-processing" containerID="cri-o://ffec03e3baa59e9d6940b2cac130304f9a877f41a38bb55a1bb5d3ee9ce0f2d5" gracePeriod=30
Sep 17 14:24:58.313653 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:58.313626    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:58.412398 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:58.412370    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:58.436555 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:58.436522    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d9b17d-5j4h4"]
Sep 17 14:24:58.438283 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:58.438258    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d9b17d-5j4h4"]
Sep 17 14:24:58.525547 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:58.525385    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:58.535647 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:58.535317    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d9b17d-67659db75d-fl9f9"
Sep 17 14:24:58.652568 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:58.652518    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87759bfd-b7d8-45e0-b5f0-8d5c081abaf3-plugins-config-volume\") pod \"87759bfd-b7d8-45e0-b5f0-8d5c081abaf3\" (UID: \"87759bfd-b7d8-45e0-b5f0-8d5c081abaf3\") "
Sep 17 14:24:58.653478 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:58.653392    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2rfxq\" (UniqueName: \"kubernetes.io/projected/87759bfd-b7d8-45e0-b5f0-8d5c081abaf3-kube-api-access-2rfxq\") pod \"87759bfd-b7d8-45e0-b5f0-8d5c081abaf3\" (UID: \"87759bfd-b7d8-45e0-b5f0-8d5c081abaf3\") "
Sep 17 14:24:58.653721 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:58.653323    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87759bfd-b7d8-45e0-b5f0-8d5c081abaf3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "87759bfd-b7d8-45e0-b5f0-8d5c081abaf3" (UID: "87759bfd-b7d8-45e0-b5f0-8d5c081abaf3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:24:58.654285 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:58.654263    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87759bfd-b7d8-45e0-b5f0-8d5c081abaf3-plugins-config-volume\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:24:58.655706 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:58.655682    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87759bfd-b7d8-45e0-b5f0-8d5c081abaf3-kube-api-access-2rfxq" (OuterVolumeSpecName: "kube-api-access-2rfxq") pod "87759bfd-b7d8-45e0-b5f0-8d5c081abaf3" (UID: "87759bfd-b7d8-45e0-b5f0-8d5c081abaf3"). InnerVolumeSpecName "kube-api-access-2rfxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:24:58.755181 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:58.755116    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rfxq\" (UniqueName: \"kubernetes.io/projected/87759bfd-b7d8-45e0-b5f0-8d5c081abaf3-kube-api-access-2rfxq\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:24:58.797785 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:58.797755    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:58.890538 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:58.890506    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:59.089698 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:59.089669    2560 generic.go:356] "Generic (PLEG): container finished" podID="87759bfd-b7d8-45e0-b5f0-8d5c081abaf3" containerID="ffec03e3baa59e9d6940b2cac130304f9a877f41a38bb55a1bb5d3ee9ce0f2d5" exitCode=0
Sep 17 14:24:59.089885 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:59.089709    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d9b17d-67659db75d-fl9f9"
Sep 17 14:24:59.089885 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:59.089753    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d9b17d-67659db75d-fl9f9" event={"ID":"87759bfd-b7d8-45e0-b5f0-8d5c081abaf3","Type":"ContainerDied","Data":"ffec03e3baa59e9d6940b2cac130304f9a877f41a38bb55a1bb5d3ee9ce0f2d5"}
Sep 17 14:24:59.089885 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:59.089805    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d9b17d-67659db75d-fl9f9" event={"ID":"87759bfd-b7d8-45e0-b5f0-8d5c081abaf3","Type":"ContainerDied","Data":"8c57355051f3cb23417bde5b45d9309eb426cc727e63a62589e1180199fce48f"}
Sep 17 14:24:59.089885 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:59.089831    2560 scope.go:117] "RemoveContainer" containerID="ffec03e3baa59e9d6940b2cac130304f9a877f41a38bb55a1bb5d3ee9ce0f2d5"
Sep 17 14:24:59.100112 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:59.100098    2560 scope.go:117] "RemoveContainer" containerID="ffec03e3baa59e9d6940b2cac130304f9a877f41a38bb55a1bb5d3ee9ce0f2d5"
Sep 17 14:24:59.100363 ip-10-0-129-158 kubenswrapper[2560]: E0917 14:24:59.100345    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffec03e3baa59e9d6940b2cac130304f9a877f41a38bb55a1bb5d3ee9ce0f2d5\": container with ID starting with ffec03e3baa59e9d6940b2cac130304f9a877f41a38bb55a1bb5d3ee9ce0f2d5 not found: ID does not exist" containerID="ffec03e3baa59e9d6940b2cac130304f9a877f41a38bb55a1bb5d3ee9ce0f2d5"
Sep 17 14:24:59.100407 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:59.100372    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffec03e3baa59e9d6940b2cac130304f9a877f41a38bb55a1bb5d3ee9ce0f2d5"} err="failed to get container status \"ffec03e3baa59e9d6940b2cac130304f9a877f41a38bb55a1bb5d3ee9ce0f2d5\": rpc error: code = NotFound desc = could not find container \"ffec03e3baa59e9d6940b2cac130304f9a877f41a38bb55a1bb5d3ee9ce0f2d5\": container with ID starting with ffec03e3baa59e9d6940b2cac130304f9a877f41a38bb55a1bb5d3ee9ce0f2d5 not found: ID does not exist"
Sep 17 14:24:59.120928 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:59.120632    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-d9b17d-67659db75d-fl9f9"]
Sep 17 14:24:59.121703 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:59.121683    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-d9b17d-67659db75d-fl9f9"]
Sep 17 14:24:59.570326 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:59.570291    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4341743e-5216-4b8d-9310-78acf4a761b4" path="/var/lib/kubelet/pods/4341743e-5216-4b8d-9310-78acf4a761b4/volumes"
Sep 17 14:24:59.570704 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:59.570686    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87759bfd-b7d8-45e0-b5f0-8d5c081abaf3" path="/var/lib/kubelet/pods/87759bfd-b7d8-45e0-b5f0-8d5c081abaf3/volumes"
Sep 17 14:24:59.571027 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:59.571010    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af34693e-c05d-4894-96cb-19a6523871a1" path="/var/lib/kubelet/pods/af34693e-c05d-4894-96cb-19a6523871a1/volumes"
Sep 17 14:24:59.793143 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:59.793096    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:59.836761 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:59.836689    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-d9b17d/distinct-2-w0-d9b17d-kserve-56497687d9-7x45h" podUID="6353689b-97e8-457d-b0e1-8f167deff124" containerName="main" containerID="cri-o://54d1c47ad4ebeab01653e53e688eb0def08c7e7b993eccbbb42d95f572b3ff87" gracePeriod=45
Sep 17 14:24:59.847848 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:59.847809    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-d9b17d/distinct-w0-d9b17d-kserve-85bc8c598b-cflt6" podUID="2c15de4b-4e65-4d7e-8155-55d3d9241e7b" containerName="main" containerID="cri-o://4d8971137530bc706fe961351bddda9f056790b5520bca8f1f6db28218e46d47" gracePeriod=45
Sep 17 14:24:59.848482 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:59.848446    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-d9b17d/embedding-w0-d9b17d-kserve-547f78f6c5-vhlvm" podUID="668b3d6d-a30f-4949-bd22-292b896116f6" containerName="main" containerID="cri-o://52b556f56edea34deb819d5707c0436afd6943db7b5184fef77be6b37fc944b6" gracePeriod=45
Sep 17 14:24:59.909884 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:24:59.909838    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d9b17d/distinct-2-w0-d9b17d-kserve-56497687d9-7x45h" podUID="6353689b-97e8-457d-b0e1-8f167deff124" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": dial tcp 10.132.0.89:8000: connect: connection refused"
Sep 17 14:25:00.011454 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.011178    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-d9b17d/premium-w0-d9b17d-kserve-54647796d7-4r49f" podUID="2809a4f4-2371-4513-8924-461ba4b6e5cf" containerName="main" containerID="cri-o://85438e8fed849f758191ac4225a745f60a7f9c5cc1c0124f14ab505afa347ab1" gracePeriod=45
Sep 17 14:25:00.102629 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.102353    2560 generic.go:356] "Generic (PLEG): container finished" podID="6353689b-97e8-457d-b0e1-8f167deff124" containerID="54d1c47ad4ebeab01653e53e688eb0def08c7e7b993eccbbb42d95f572b3ff87" exitCode=0
Sep 17 14:25:00.102629 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.102520    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d9b17d/distinct-2-w0-d9b17d-kserve-56497687d9-7x45h" event={"ID":"6353689b-97e8-457d-b0e1-8f167deff124","Type":"ContainerDied","Data":"54d1c47ad4ebeab01653e53e688eb0def08c7e7b993eccbbb42d95f572b3ff87"}
Sep 17 14:25:00.104834 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.104731    2560 generic.go:356] "Generic (PLEG): container finished" podID="668b3d6d-a30f-4949-bd22-292b896116f6" containerID="52b556f56edea34deb819d5707c0436afd6943db7b5184fef77be6b37fc944b6" exitCode=0
Sep 17 14:25:00.104979 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.104959    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d9b17d/embedding-w0-d9b17d-kserve-547f78f6c5-vhlvm" event={"ID":"668b3d6d-a30f-4949-bd22-292b896116f6","Type":"ContainerDied","Data":"52b556f56edea34deb819d5707c0436afd6943db7b5184fef77be6b37fc944b6"}
Sep 17 14:25:00.107371 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.107231    2560 generic.go:356] "Generic (PLEG): container finished" podID="2c15de4b-4e65-4d7e-8155-55d3d9241e7b" containerID="4d8971137530bc706fe961351bddda9f056790b5520bca8f1f6db28218e46d47" exitCode=0
Sep 17 14:25:00.107371 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.107277    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d9b17d/distinct-w0-d9b17d-kserve-85bc8c598b-cflt6" event={"ID":"2c15de4b-4e65-4d7e-8155-55d3d9241e7b","Type":"ContainerDied","Data":"4d8971137530bc706fe961351bddda9f056790b5520bca8f1f6db28218e46d47"}
Sep 17 14:25:00.131328 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.131305    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d9b17d/distinct-2-w0-d9b17d-kserve-56497687d9-7x45h"
Sep 17 14:25:00.137657 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.137635    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d9b17d/distinct-w0-d9b17d-kserve-85bc8c598b-cflt6"
Sep 17 14:25:00.151797 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.151768    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d9b17d/embedding-w0-d9b17d-kserve-547f78f6c5-vhlvm"
Sep 17 14:25:00.169555 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.169525    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c15de4b-4e65-4d7e-8155-55d3d9241e7b-model-cache\") pod \"2c15de4b-4e65-4d7e-8155-55d3d9241e7b\" (UID: \"2c15de4b-4e65-4d7e-8155-55d3d9241e7b\") "
Sep 17 14:25:00.169695 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.169633    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6353689b-97e8-457d-b0e1-8f167deff124-tls-certs\") pod \"6353689b-97e8-457d-b0e1-8f167deff124\" (UID: \"6353689b-97e8-457d-b0e1-8f167deff124\") "
Sep 17 14:25:00.169695 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.169691    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c15de4b-4e65-4d7e-8155-55d3d9241e7b-dshm\") pod \"2c15de4b-4e65-4d7e-8155-55d3d9241e7b\" (UID: \"2c15de4b-4e65-4d7e-8155-55d3d9241e7b\") "
Sep 17 14:25:00.169824 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.169735    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6353689b-97e8-457d-b0e1-8f167deff124-home\") pod \"6353689b-97e8-457d-b0e1-8f167deff124\" (UID: \"6353689b-97e8-457d-b0e1-8f167deff124\") "
Sep 17 14:25:00.169824 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.169811    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6353689b-97e8-457d-b0e1-8f167deff124-dshm\") pod \"6353689b-97e8-457d-b0e1-8f167deff124\" (UID: \"6353689b-97e8-457d-b0e1-8f167deff124\") "
Sep 17 14:25:00.169934 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.169838    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lpv5t\" (UniqueName: \"kubernetes.io/projected/2c15de4b-4e65-4d7e-8155-55d3d9241e7b-kube-api-access-lpv5t\") pod \"2c15de4b-4e65-4d7e-8155-55d3d9241e7b\" (UID: \"2c15de4b-4e65-4d7e-8155-55d3d9241e7b\") "
Sep 17 14:25:00.169934 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.169899    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c15de4b-4e65-4d7e-8155-55d3d9241e7b-tls-certs\") pod \"2c15de4b-4e65-4d7e-8155-55d3d9241e7b\" (UID: \"2c15de4b-4e65-4d7e-8155-55d3d9241e7b\") "
Sep 17 14:25:00.169934 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.169927    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6353689b-97e8-457d-b0e1-8f167deff124-model-cache\") pod \"6353689b-97e8-457d-b0e1-8f167deff124\" (UID: \"6353689b-97e8-457d-b0e1-8f167deff124\") "
Sep 17 14:25:00.170070 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.169951    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6353689b-97e8-457d-b0e1-8f167deff124-tmp-dir\") pod \"6353689b-97e8-457d-b0e1-8f167deff124\" (UID: \"6353689b-97e8-457d-b0e1-8f167deff124\") "
Sep 17 14:25:00.170070 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.170022    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c15de4b-4e65-4d7e-8155-55d3d9241e7b-home\") pod \"2c15de4b-4e65-4d7e-8155-55d3d9241e7b\" (UID: \"2c15de4b-4e65-4d7e-8155-55d3d9241e7b\") "
Sep 17 14:25:00.170070 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.170050    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c15de4b-4e65-4d7e-8155-55d3d9241e7b-tmp-dir\") pod \"2c15de4b-4e65-4d7e-8155-55d3d9241e7b\" (UID: \"2c15de4b-4e65-4d7e-8155-55d3d9241e7b\") "
Sep 17 14:25:00.170728 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.170686    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6353689b-97e8-457d-b0e1-8f167deff124-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6353689b-97e8-457d-b0e1-8f167deff124" (UID: "6353689b-97e8-457d-b0e1-8f167deff124"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:25:00.171041 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.170147    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tjxj5\" (UniqueName: \"kubernetes.io/projected/6353689b-97e8-457d-b0e1-8f167deff124-kube-api-access-tjxj5\") pod \"6353689b-97e8-457d-b0e1-8f167deff124\" (UID: \"6353689b-97e8-457d-b0e1-8f167deff124\") "
Sep 17 14:25:00.171625 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.171601    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6353689b-97e8-457d-b0e1-8f167deff124-tmp-dir\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:25:00.172836 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.172188    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6353689b-97e8-457d-b0e1-8f167deff124-model-cache" (OuterVolumeSpecName: "model-cache") pod "6353689b-97e8-457d-b0e1-8f167deff124" (UID: "6353689b-97e8-457d-b0e1-8f167deff124"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:25:00.172836 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.172383    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c15de4b-4e65-4d7e-8155-55d3d9241e7b-model-cache" (OuterVolumeSpecName: "model-cache") pod "2c15de4b-4e65-4d7e-8155-55d3d9241e7b" (UID: "2c15de4b-4e65-4d7e-8155-55d3d9241e7b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:25:00.172836 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.172391    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c15de4b-4e65-4d7e-8155-55d3d9241e7b-home" (OuterVolumeSpecName: "home") pod "2c15de4b-4e65-4d7e-8155-55d3d9241e7b" (UID: "2c15de4b-4e65-4d7e-8155-55d3d9241e7b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:25:00.172836 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.172587    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c15de4b-4e65-4d7e-8155-55d3d9241e7b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2c15de4b-4e65-4d7e-8155-55d3d9241e7b" (UID: "2c15de4b-4e65-4d7e-8155-55d3d9241e7b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:25:00.172836 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.172594    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6353689b-97e8-457d-b0e1-8f167deff124-home" (OuterVolumeSpecName: "home") pod "6353689b-97e8-457d-b0e1-8f167deff124" (UID: "6353689b-97e8-457d-b0e1-8f167deff124"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:25:00.176119 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.175972    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c15de4b-4e65-4d7e-8155-55d3d9241e7b-dshm" (OuterVolumeSpecName: "dshm") pod "2c15de4b-4e65-4d7e-8155-55d3d9241e7b" (UID: "2c15de4b-4e65-4d7e-8155-55d3d9241e7b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:25:00.176119 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.176078    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6353689b-97e8-457d-b0e1-8f167deff124-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6353689b-97e8-457d-b0e1-8f167deff124" (UID: "6353689b-97e8-457d-b0e1-8f167deff124"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:25:00.176390 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.176356    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c15de4b-4e65-4d7e-8155-55d3d9241e7b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2c15de4b-4e65-4d7e-8155-55d3d9241e7b" (UID: "2c15de4b-4e65-4d7e-8155-55d3d9241e7b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:25:00.176904 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.176874    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6353689b-97e8-457d-b0e1-8f167deff124-kube-api-access-tjxj5" (OuterVolumeSpecName: "kube-api-access-tjxj5") pod "6353689b-97e8-457d-b0e1-8f167deff124" (UID: "6353689b-97e8-457d-b0e1-8f167deff124"). InnerVolumeSpecName "kube-api-access-tjxj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:25:00.177586 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.177555    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6353689b-97e8-457d-b0e1-8f167deff124-dshm" (OuterVolumeSpecName: "dshm") pod "6353689b-97e8-457d-b0e1-8f167deff124" (UID: "6353689b-97e8-457d-b0e1-8f167deff124"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:25:00.177664 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.177645    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c15de4b-4e65-4d7e-8155-55d3d9241e7b-kube-api-access-lpv5t" (OuterVolumeSpecName: "kube-api-access-lpv5t") pod "2c15de4b-4e65-4d7e-8155-55d3d9241e7b" (UID: "2c15de4b-4e65-4d7e-8155-55d3d9241e7b"). InnerVolumeSpecName "kube-api-access-lpv5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:25:00.272544 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.272510    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/668b3d6d-a30f-4949-bd22-292b896116f6-tls-certs\") pod \"668b3d6d-a30f-4949-bd22-292b896116f6\" (UID: \"668b3d6d-a30f-4949-bd22-292b896116f6\") "
Sep 17 14:25:00.272718 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.272582    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tdgqd\" (UniqueName: \"kubernetes.io/projected/668b3d6d-a30f-4949-bd22-292b896116f6-kube-api-access-tdgqd\") pod \"668b3d6d-a30f-4949-bd22-292b896116f6\" (UID: \"668b3d6d-a30f-4949-bd22-292b896116f6\") "
Sep 17 14:25:00.272718 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.272676    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/668b3d6d-a30f-4949-bd22-292b896116f6-tmp-dir\") pod \"668b3d6d-a30f-4949-bd22-292b896116f6\" (UID: \"668b3d6d-a30f-4949-bd22-292b896116f6\") "
Sep 17 14:25:00.272827 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.272721    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/668b3d6d-a30f-4949-bd22-292b896116f6-dshm\") pod \"668b3d6d-a30f-4949-bd22-292b896116f6\" (UID: \"668b3d6d-a30f-4949-bd22-292b896116f6\") "
Sep 17 14:25:00.272827 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.272779    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/668b3d6d-a30f-4949-bd22-292b896116f6-home\") pod \"668b3d6d-a30f-4949-bd22-292b896116f6\" (UID: \"668b3d6d-a30f-4949-bd22-292b896116f6\") "
Sep 17 14:25:00.272827 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.272802    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/668b3d6d-a30f-4949-bd22-292b896116f6-model-cache\") pod \"668b3d6d-a30f-4949-bd22-292b896116f6\" (UID: \"668b3d6d-a30f-4949-bd22-292b896116f6\") "
Sep 17 14:25:00.272998 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.272930    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/668b3d6d-a30f-4949-bd22-292b896116f6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "668b3d6d-a30f-4949-bd22-292b896116f6" (UID: "668b3d6d-a30f-4949-bd22-292b896116f6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:25:00.273054 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.273019    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/668b3d6d-a30f-4949-bd22-292b896116f6-home" (OuterVolumeSpecName: "home") pod "668b3d6d-a30f-4949-bd22-292b896116f6" (UID: "668b3d6d-a30f-4949-bd22-292b896116f6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:25:00.273208 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.273189    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/668b3d6d-a30f-4949-bd22-292b896116f6-model-cache" (OuterVolumeSpecName: "model-cache") pod "668b3d6d-a30f-4949-bd22-292b896116f6" (UID: "668b3d6d-a30f-4949-bd22-292b896116f6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:25:00.273374 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.273360    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6353689b-97e8-457d-b0e1-8f167deff124-home\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:25:00.273444 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.273375    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6353689b-97e8-457d-b0e1-8f167deff124-dshm\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:25:00.273444 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.273385    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpv5t\" (UniqueName: \"kubernetes.io/projected/2c15de4b-4e65-4d7e-8155-55d3d9241e7b-kube-api-access-lpv5t\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:25:00.273444 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.273395    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c15de4b-4e65-4d7e-8155-55d3d9241e7b-tls-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:25:00.273444 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.273403    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/668b3d6d-a30f-4949-bd22-292b896116f6-home\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:25:00.273444 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.273410    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6353689b-97e8-457d-b0e1-8f167deff124-model-cache\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:25:00.273444 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.273417    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/668b3d6d-a30f-4949-bd22-292b896116f6-model-cache\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:25:00.273444 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.273424    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c15de4b-4e65-4d7e-8155-55d3d9241e7b-home\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:25:00.273444 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.273432    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c15de4b-4e65-4d7e-8155-55d3d9241e7b-tmp-dir\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:25:00.273444 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.273440    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjxj5\" (UniqueName: \"kubernetes.io/projected/6353689b-97e8-457d-b0e1-8f167deff124-kube-api-access-tjxj5\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:25:00.273444 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.273448    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c15de4b-4e65-4d7e-8155-55d3d9241e7b-model-cache\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:25:00.273988 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.273456    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6353689b-97e8-457d-b0e1-8f167deff124-tls-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:25:00.273988 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.273482    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c15de4b-4e65-4d7e-8155-55d3d9241e7b-dshm\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:25:00.273988 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.273490    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/668b3d6d-a30f-4949-bd22-292b896116f6-tmp-dir\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:25:00.274731 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.274713    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/668b3d6d-a30f-4949-bd22-292b896116f6-dshm" (OuterVolumeSpecName: "dshm") pod "668b3d6d-a30f-4949-bd22-292b896116f6" (UID: "668b3d6d-a30f-4949-bd22-292b896116f6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:25:00.274824 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.274797    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/668b3d6d-a30f-4949-bd22-292b896116f6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "668b3d6d-a30f-4949-bd22-292b896116f6" (UID: "668b3d6d-a30f-4949-bd22-292b896116f6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:25:00.275031 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.275011    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/668b3d6d-a30f-4949-bd22-292b896116f6-kube-api-access-tdgqd" (OuterVolumeSpecName: "kube-api-access-tdgqd") pod "668b3d6d-a30f-4949-bd22-292b896116f6" (UID: "668b3d6d-a30f-4949-bd22-292b896116f6"). InnerVolumeSpecName "kube-api-access-tdgqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:25:00.289595 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.289575    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d9b17d/premium-w0-d9b17d-kserve-54647796d7-4r49f"
Sep 17 14:25:00.374192 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.374112    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2809a4f4-2371-4513-8924-461ba4b6e5cf-tls-certs\") pod \"2809a4f4-2371-4513-8924-461ba4b6e5cf\" (UID: \"2809a4f4-2371-4513-8924-461ba4b6e5cf\") "
Sep 17 14:25:00.374192 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.374162    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2809a4f4-2371-4513-8924-461ba4b6e5cf-home\") pod \"2809a4f4-2371-4513-8924-461ba4b6e5cf\" (UID: \"2809a4f4-2371-4513-8924-461ba4b6e5cf\") "
Sep 17 14:25:00.374358 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.374198    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2809a4f4-2371-4513-8924-461ba4b6e5cf-tmp-dir\") pod \"2809a4f4-2371-4513-8924-461ba4b6e5cf\" (UID: \"2809a4f4-2371-4513-8924-461ba4b6e5cf\") "
Sep 17 14:25:00.374358 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.374229    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-58qzg\" (UniqueName: \"kubernetes.io/projected/2809a4f4-2371-4513-8924-461ba4b6e5cf-kube-api-access-58qzg\") pod \"2809a4f4-2371-4513-8924-461ba4b6e5cf\" (UID: \"2809a4f4-2371-4513-8924-461ba4b6e5cf\") "
Sep 17 14:25:00.374358 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.374261    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2809a4f4-2371-4513-8924-461ba4b6e5cf-dshm\") pod \"2809a4f4-2371-4513-8924-461ba4b6e5cf\" (UID: \"2809a4f4-2371-4513-8924-461ba4b6e5cf\") "
Sep 17 14:25:00.374358 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.374345    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2809a4f4-2371-4513-8924-461ba4b6e5cf-model-cache\") pod \"2809a4f4-2371-4513-8924-461ba4b6e5cf\" (UID: \"2809a4f4-2371-4513-8924-461ba4b6e5cf\") "
Sep 17 14:25:00.374570 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.374484    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2809a4f4-2371-4513-8924-461ba4b6e5cf-home" (OuterVolumeSpecName: "home") pod "2809a4f4-2371-4513-8924-461ba4b6e5cf" (UID: "2809a4f4-2371-4513-8924-461ba4b6e5cf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:25:00.374570 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.374520    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2809a4f4-2371-4513-8924-461ba4b6e5cf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2809a4f4-2371-4513-8924-461ba4b6e5cf" (UID: "2809a4f4-2371-4513-8924-461ba4b6e5cf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:25:00.374703 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.374669    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2809a4f4-2371-4513-8924-461ba4b6e5cf-model-cache" (OuterVolumeSpecName: "model-cache") pod "2809a4f4-2371-4513-8924-461ba4b6e5cf" (UID: "2809a4f4-2371-4513-8924-461ba4b6e5cf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:25:00.374864 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.374847    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/668b3d6d-a30f-4949-bd22-292b896116f6-tls-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:25:00.374937 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.374866    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2809a4f4-2371-4513-8924-461ba4b6e5cf-model-cache\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:25:00.374937 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.374879    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdgqd\" (UniqueName: \"kubernetes.io/projected/668b3d6d-a30f-4949-bd22-292b896116f6-kube-api-access-tdgqd\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:25:00.374937 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.374893    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/668b3d6d-a30f-4949-bd22-292b896116f6-dshm\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:25:00.374937 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.374906    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2809a4f4-2371-4513-8924-461ba4b6e5cf-home\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:25:00.374937 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.374916    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2809a4f4-2371-4513-8924-461ba4b6e5cf-tmp-dir\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:25:00.376308 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.376283    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2809a4f4-2371-4513-8924-461ba4b6e5cf-dshm" (OuterVolumeSpecName: "dshm") pod "2809a4f4-2371-4513-8924-461ba4b6e5cf" (UID: "2809a4f4-2371-4513-8924-461ba4b6e5cf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:25:00.376413 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.376311    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2809a4f4-2371-4513-8924-461ba4b6e5cf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2809a4f4-2371-4513-8924-461ba4b6e5cf" (UID: "2809a4f4-2371-4513-8924-461ba4b6e5cf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:25:00.376413 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.376353    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2809a4f4-2371-4513-8924-461ba4b6e5cf-kube-api-access-58qzg" (OuterVolumeSpecName: "kube-api-access-58qzg") pod "2809a4f4-2371-4513-8924-461ba4b6e5cf" (UID: "2809a4f4-2371-4513-8924-461ba4b6e5cf"). InnerVolumeSpecName "kube-api-access-58qzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:25:00.476510 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.476455    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58qzg\" (UniqueName: \"kubernetes.io/projected/2809a4f4-2371-4513-8924-461ba4b6e5cf-kube-api-access-58qzg\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:25:00.476510 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.476506    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2809a4f4-2371-4513-8924-461ba4b6e5cf-dshm\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:25:00.476673 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.476520    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2809a4f4-2371-4513-8924-461ba4b6e5cf-tls-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:25:00.703921 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:00.703823    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:01.115584 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:01.115550    2560 generic.go:356] "Generic (PLEG): container finished" podID="2809a4f4-2371-4513-8924-461ba4b6e5cf" containerID="85438e8fed849f758191ac4225a745f60a7f9c5cc1c0124f14ab505afa347ab1" exitCode=0
Sep 17 14:25:01.115584 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:01.115581    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d9b17d/premium-w0-d9b17d-kserve-54647796d7-4r49f"
Sep 17 14:25:01.115811 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:01.115631    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d9b17d/premium-w0-d9b17d-kserve-54647796d7-4r49f" event={"ID":"2809a4f4-2371-4513-8924-461ba4b6e5cf","Type":"ContainerDied","Data":"85438e8fed849f758191ac4225a745f60a7f9c5cc1c0124f14ab505afa347ab1"}
Sep 17 14:25:01.115811 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:01.115675    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d9b17d/premium-w0-d9b17d-kserve-54647796d7-4r49f" event={"ID":"2809a4f4-2371-4513-8924-461ba4b6e5cf","Type":"ContainerDied","Data":"01ae4de72d17a88d455caebad13c9d7133c415f623a26431b07c27a7f933a5a1"}
Sep 17 14:25:01.115811 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:01.115699    2560 scope.go:117] "RemoveContainer" containerID="85438e8fed849f758191ac4225a745f60a7f9c5cc1c0124f14ab505afa347ab1"
Sep 17 14:25:01.117341 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:01.117325    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d9b17d/distinct-w0-d9b17d-kserve-85bc8c598b-cflt6"
Sep 17 14:25:01.117437 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:01.117348    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d9b17d/distinct-w0-d9b17d-kserve-85bc8c598b-cflt6" event={"ID":"2c15de4b-4e65-4d7e-8155-55d3d9241e7b","Type":"ContainerDied","Data":"0bc66c58f6b714b9c9bfd5d6765becb9759a4ece624737e53d81a08d42b24c35"}
Sep 17 14:25:01.118992 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:01.118963    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d9b17d/distinct-2-w0-d9b17d-kserve-56497687d9-7x45h"
Sep 17 14:25:01.119094 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:01.118992    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d9b17d/distinct-2-w0-d9b17d-kserve-56497687d9-7x45h" event={"ID":"6353689b-97e8-457d-b0e1-8f167deff124","Type":"ContainerDied","Data":"1d70a38a4e0ce2aab3fcd58b4471a07659424feda8b7cc3a5eac2c8456804118"}
Sep 17 14:25:01.120568 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:01.120548    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d9b17d/embedding-w0-d9b17d-kserve-547f78f6c5-vhlvm"
Sep 17 14:25:01.120684 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:01.120623    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d9b17d/embedding-w0-d9b17d-kserve-547f78f6c5-vhlvm" event={"ID":"668b3d6d-a30f-4949-bd22-292b896116f6","Type":"ContainerDied","Data":"276b70d62c97af83dc310588729186ba7bc0b725ec839272b8455ccbe9d37c75"}
Sep 17 14:25:01.130094 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:01.130075    2560 scope.go:117] "RemoveContainer" containerID="85438e8fed849f758191ac4225a745f60a7f9c5cc1c0124f14ab505afa347ab1"
Sep 17 14:25:01.130357 ip-10-0-129-158 kubenswrapper[2560]: E0917 14:25:01.130336    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85438e8fed849f758191ac4225a745f60a7f9c5cc1c0124f14ab505afa347ab1\": container with ID starting with 85438e8fed849f758191ac4225a745f60a7f9c5cc1c0124f14ab505afa347ab1 not found: ID does not exist" containerID="85438e8fed849f758191ac4225a745f60a7f9c5cc1c0124f14ab505afa347ab1"
Sep 17 14:25:01.130412 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:01.130363    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85438e8fed849f758191ac4225a745f60a7f9c5cc1c0124f14ab505afa347ab1"} err="failed to get container status \"85438e8fed849f758191ac4225a745f60a7f9c5cc1c0124f14ab505afa347ab1\": rpc error: code = NotFound desc = could not find container \"85438e8fed849f758191ac4225a745f60a7f9c5cc1c0124f14ab505afa347ab1\": container with ID starting with 85438e8fed849f758191ac4225a745f60a7f9c5cc1c0124f14ab505afa347ab1 not found: ID does not exist"
Sep 17 14:25:01.130412 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:01.130379    2560 scope.go:117] "RemoveContainer" containerID="4d8971137530bc706fe961351bddda9f056790b5520bca8f1f6db28218e46d47"
Sep 17 14:25:01.139293 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:01.139273    2560 scope.go:117] "RemoveContainer" containerID="54d1c47ad4ebeab01653e53e688eb0def08c7e7b993eccbbb42d95f572b3ff87"
Sep 17 14:25:01.147643 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:01.147625    2560 scope.go:117] "RemoveContainer" containerID="52b556f56edea34deb819d5707c0436afd6943db7b5184fef77be6b37fc944b6"
Sep 17 14:25:01.181159 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:01.181134    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d9b17d/distinct-w0-d9b17d-kserve-85bc8c598b-cflt6"]
Sep 17 14:25:01.184851 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:01.184828    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-d9b17d/distinct-w0-d9b17d-kserve-85bc8c598b-cflt6"]
Sep 17 14:25:01.222121 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:01.222096    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d9b17d/distinct-2-w0-d9b17d-kserve-56497687d9-7x45h"]
Sep 17 14:25:01.228831 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:01.228809    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-d9b17d/distinct-2-w0-d9b17d-kserve-56497687d9-7x45h"]
Sep 17 14:25:01.251304 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:01.251274    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d9b17d/embedding-w0-d9b17d-kserve-547f78f6c5-vhlvm"]
Sep 17 14:25:01.261775 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:01.261751    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-d9b17d/embedding-w0-d9b17d-kserve-547f78f6c5-vhlvm"]
Sep 17 14:25:01.292757 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:01.292729    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d9b17d/premium-w0-d9b17d-kserve-54647796d7-4r49f"]
Sep 17 14:25:01.295358 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:01.295336    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-d9b17d/premium-w0-d9b17d-kserve-54647796d7-4r49f"]
Sep 17 14:25:01.570909 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:01.570880    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2809a4f4-2371-4513-8924-461ba4b6e5cf" path="/var/lib/kubelet/pods/2809a4f4-2371-4513-8924-461ba4b6e5cf/volumes"
Sep 17 14:25:01.571346 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:01.571330    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c15de4b-4e65-4d7e-8155-55d3d9241e7b" path="/var/lib/kubelet/pods/2c15de4b-4e65-4d7e-8155-55d3d9241e7b/volumes"
Sep 17 14:25:01.571765 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:01.571751    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6353689b-97e8-457d-b0e1-8f167deff124" path="/var/lib/kubelet/pods/6353689b-97e8-457d-b0e1-8f167deff124/volumes"
Sep 17 14:25:01.572296 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:01.572278    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="668b3d6d-a30f-4949-bd22-292b896116f6" path="/var/lib/kubelet/pods/668b3d6d-a30f-4949-bd22-292b896116f6/volumes"
Sep 17 14:25:01.760078 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:01.760043    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:02.472390 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:02.472337    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:02.567939 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:02.567903    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:02.650438 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:02.650398    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:02.763397 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:02.763310    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:02.924873 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:02.924837    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:02.937345 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:02.937317    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6fce83-2tt5z"]
Sep 17 14:25:02.947646 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:02.947624    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6fce83-2tt5z"]
Sep 17 14:25:03.029228 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:03.029053    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:03.248331 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:03.248300    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:03.570863 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:03.570834    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b625e8de-13a9-4b72-b87f-079af3240401" path="/var/lib/kubelet/pods/b625e8de-13a9-4b72-b87f-079af3240401/volumes"
Sep 17 14:25:03.998770 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:03.998681    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:04.784442 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:04.784405    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:05.581144 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:05.581108    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:06.412811 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:06.412773    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:06.591915 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:06.591887    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:06.753142 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:06.753063    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:06.856856 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:06.856818    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s"]
Sep 17 14:25:06.857125 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:06.857041    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s" podUID="47668435-ed43-4f78-a71b-ceb972a66e25" containerName="istio-proxy" containerID="cri-o://ad05498cd1fe227512c4c17430f86d321638ff7a9eda259aaedefed815f08517" gracePeriod=30
Sep 17 14:25:06.881453 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:06.881426    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:07.209369 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:07.209332    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:08.100686 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:08.100646    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:08.846034 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:08.846001    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:10.288082 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:10.288050    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:10.371867 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:10.371835    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:10.624868 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:10.624837    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:10.934178 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:10.934103    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a77c87-7456v"]
Sep 17 14:25:10.934559 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:10.934547    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2809a4f4-2371-4513-8924-461ba4b6e5cf" containerName="main"
Sep 17 14:25:10.934616 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:10.934561    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2809a4f4-2371-4513-8924-461ba4b6e5cf" containerName="main"
Sep 17 14:25:10.934616 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:10.934576    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87759bfd-b7d8-45e0-b5f0-8d5c081abaf3" containerName="payload-pre-processing"
Sep 17 14:25:10.934616 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:10.934582    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="87759bfd-b7d8-45e0-b5f0-8d5c081abaf3" containerName="payload-pre-processing"
Sep 17 14:25:10.934616 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:10.934600    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b625e8de-13a9-4b72-b87f-079af3240401" containerName="revoke-keys"
Sep 17 14:25:10.934616 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:10.934605    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b625e8de-13a9-4b72-b87f-079af3240401" containerName="revoke-keys"
Sep 17 14:25:10.934616 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:10.934613    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c15de4b-4e65-4d7e-8155-55d3d9241e7b" containerName="main"
Sep 17 14:25:10.934616 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:10.934618    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c15de4b-4e65-4d7e-8155-55d3d9241e7b" containerName="main"
Sep 17 14:25:10.934822 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:10.934626    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="668b3d6d-a30f-4949-bd22-292b896116f6" containerName="main"
Sep 17 14:25:10.934822 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:10.934631    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="668b3d6d-a30f-4949-bd22-292b896116f6" containerName="main"
Sep 17 14:25:10.934822 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:10.934643    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6353689b-97e8-457d-b0e1-8f167deff124" containerName="main"
Sep 17 14:25:10.934822 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:10.934648    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6353689b-97e8-457d-b0e1-8f167deff124" containerName="main"
Sep 17 14:25:10.934822 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:10.934709    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="668b3d6d-a30f-4949-bd22-292b896116f6" containerName="main"
Sep 17 14:25:10.934822 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:10.934718    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c15de4b-4e65-4d7e-8155-55d3d9241e7b" containerName="main"
Sep 17 14:25:10.934822 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:10.934725    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="87759bfd-b7d8-45e0-b5f0-8d5c081abaf3" containerName="payload-pre-processing"
Sep 17 14:25:10.934822 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:10.934731    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6353689b-97e8-457d-b0e1-8f167deff124" containerName="main"
Sep 17 14:25:10.934822 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:10.934736    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="2809a4f4-2371-4513-8924-461ba4b6e5cf" containerName="main"
Sep 17 14:25:10.934822 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:10.934742    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b625e8de-13a9-4b72-b87f-079af3240401" containerName="revoke-keys"
Sep 17 14:25:10.937876 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:10.937861    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a77c87-7456v"
Sep 17 14:25:10.940139 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:10.940113    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 14:25:10.940274 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:10.940215    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rp9ww\""
Sep 17 14:25:10.947323 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:10.947301    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a77c87-7456v"]
Sep 17 14:25:11.085366 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:11.085333    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4610e90-8580-4fcc-8c74-165ae57d2144-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a77c87-7456v\" (UID: \"d4610e90-8580-4fcc-8c74-165ae57d2144\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a77c87-7456v"
Sep 17 14:25:11.086215 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:11.086190    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:11.186965 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:11.186886    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4610e90-8580-4fcc-8c74-165ae57d2144-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a77c87-7456v\" (UID: \"d4610e90-8580-4fcc-8c74-165ae57d2144\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a77c87-7456v"
Sep 17 14:25:11.187501 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:11.187484    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4610e90-8580-4fcc-8c74-165ae57d2144-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a77c87-7456v\" (UID: \"d4610e90-8580-4fcc-8c74-165ae57d2144\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a77c87-7456v"
Sep 17 14:25:11.250262 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:11.250233    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a77c87-7456v"
Sep 17 14:25:11.535698 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:11.535672    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a77c87-7456v"]
Sep 17 14:25:11.537198 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:25:11.537172    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4610e90_8580_4fcc_8c74_165ae57d2144.slice/crio-c2bd14171880da4130f536a92930019b084ca7df953eb1f9ad1934db5ec2a287 WatchSource:0}: Error finding container c2bd14171880da4130f536a92930019b084ca7df953eb1f9ad1934db5ec2a287: Status 404 returned error can't find the container with id c2bd14171880da4130f536a92930019b084ca7df953eb1f9ad1934db5ec2a287
Sep 17 14:25:12.094382 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:12.094357    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s"
Sep 17 14:25:12.111313 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:12.111285    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:12.196511 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:12.196414    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47668435-ed43-4f78-a71b-ceb972a66e25-workload-socket\") pod \"47668435-ed43-4f78-a71b-ceb972a66e25\" (UID: \"47668435-ed43-4f78-a71b-ceb972a66e25\") "
Sep 17 14:25:12.196680 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:12.196554    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47668435-ed43-4f78-a71b-ceb972a66e25-credential-socket\") pod \"47668435-ed43-4f78-a71b-ceb972a66e25\" (UID: \"47668435-ed43-4f78-a71b-ceb972a66e25\") "
Sep 17 14:25:12.196680 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:12.196592    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47668435-ed43-4f78-a71b-ceb972a66e25-istio-envoy\") pod \"47668435-ed43-4f78-a71b-ceb972a66e25\" (UID: \"47668435-ed43-4f78-a71b-ceb972a66e25\") "
Sep 17 14:25:12.196680 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:12.196641    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47668435-ed43-4f78-a71b-ceb972a66e25-istiod-ca-cert\") pod \"47668435-ed43-4f78-a71b-ceb972a66e25\" (UID: \"47668435-ed43-4f78-a71b-ceb972a66e25\") "
Sep 17 14:25:12.196680 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:12.196666    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47668435-ed43-4f78-a71b-ceb972a66e25-istio-data\") pod \"47668435-ed43-4f78-a71b-ceb972a66e25\" (UID: \"47668435-ed43-4f78-a71b-ceb972a66e25\") "
Sep 17 14:25:12.196898 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:12.196716    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47668435-ed43-4f78-a71b-ceb972a66e25-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "47668435-ed43-4f78-a71b-ceb972a66e25" (UID: "47668435-ed43-4f78-a71b-ceb972a66e25"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:25:12.196898 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:12.196757    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lztr9\" (UniqueName: \"kubernetes.io/projected/47668435-ed43-4f78-a71b-ceb972a66e25-kube-api-access-lztr9\") pod \"47668435-ed43-4f78-a71b-ceb972a66e25\" (UID: \"47668435-ed43-4f78-a71b-ceb972a66e25\") "
Sep 17 14:25:12.196898 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:12.196802    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47668435-ed43-4f78-a71b-ceb972a66e25-workload-certs\") pod \"47668435-ed43-4f78-a71b-ceb972a66e25\" (UID: \"47668435-ed43-4f78-a71b-ceb972a66e25\") "
Sep 17 14:25:12.197059 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:12.196918    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47668435-ed43-4f78-a71b-ceb972a66e25-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "47668435-ed43-4f78-a71b-ceb972a66e25" (UID: "47668435-ed43-4f78-a71b-ceb972a66e25"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:25:12.197059 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:12.196923    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47668435-ed43-4f78-a71b-ceb972a66e25-istio-podinfo\") pod \"47668435-ed43-4f78-a71b-ceb972a66e25\" (UID: \"47668435-ed43-4f78-a71b-ceb972a66e25\") "
Sep 17 14:25:12.197059 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:12.196967    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47668435-ed43-4f78-a71b-ceb972a66e25-istio-data" (OuterVolumeSpecName: "istio-data") pod "47668435-ed43-4f78-a71b-ceb972a66e25" (UID: "47668435-ed43-4f78-a71b-ceb972a66e25"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:25:12.197059 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:12.196996    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47668435-ed43-4f78-a71b-ceb972a66e25-istio-token\") pod \"47668435-ed43-4f78-a71b-ceb972a66e25\" (UID: \"47668435-ed43-4f78-a71b-ceb972a66e25\") "
Sep 17 14:25:12.197244 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:12.197083    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47668435-ed43-4f78-a71b-ceb972a66e25-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "47668435-ed43-4f78-a71b-ceb972a66e25" (UID: "47668435-ed43-4f78-a71b-ceb972a66e25"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:25:12.197244 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:12.197137    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47668435-ed43-4f78-a71b-ceb972a66e25-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "47668435-ed43-4f78-a71b-ceb972a66e25" (UID: "47668435-ed43-4f78-a71b-ceb972a66e25"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:25:12.197426 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:12.197402    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47668435-ed43-4f78-a71b-ceb972a66e25-istiod-ca-cert\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:25:12.197550 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:12.197430    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47668435-ed43-4f78-a71b-ceb972a66e25-istio-data\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:25:12.197550 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:12.197444    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47668435-ed43-4f78-a71b-ceb972a66e25-workload-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:25:12.197550 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:12.197456    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47668435-ed43-4f78-a71b-ceb972a66e25-workload-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:25:12.197550 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:12.197483    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47668435-ed43-4f78-a71b-ceb972a66e25-credential-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:25:12.199059 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:12.199033    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47668435-ed43-4f78-a71b-ceb972a66e25-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "47668435-ed43-4f78-a71b-ceb972a66e25" (UID: "47668435-ed43-4f78-a71b-ceb972a66e25"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:25:12.199178 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:12.199102    2560 generic.go:356] "Generic (PLEG): container finished" podID="47668435-ed43-4f78-a71b-ceb972a66e25" containerID="ad05498cd1fe227512c4c17430f86d321638ff7a9eda259aaedefed815f08517" exitCode=0
Sep 17 14:25:12.199178 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:12.199157    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s"
Sep 17 14:25:12.199299 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:12.199186    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s" event={"ID":"47668435-ed43-4f78-a71b-ceb972a66e25","Type":"ContainerDied","Data":"ad05498cd1fe227512c4c17430f86d321638ff7a9eda259aaedefed815f08517"}
Sep 17 14:25:12.199299 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:12.199257    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s" event={"ID":"47668435-ed43-4f78-a71b-ceb972a66e25","Type":"ContainerDied","Data":"ab01864c94ef2561b403eac44b245743b4139dfbf354ced6ac634286b493558c"}
Sep 17 14:25:12.199380 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:12.199289    2560 scope.go:117] "RemoveContainer" containerID="ad05498cd1fe227512c4c17430f86d321638ff7a9eda259aaedefed815f08517"
Sep 17 14:25:12.199418 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:12.199381    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47668435-ed43-4f78-a71b-ceb972a66e25-istio-token" (OuterVolumeSpecName: "istio-token") pod "47668435-ed43-4f78-a71b-ceb972a66e25" (UID: "47668435-ed43-4f78-a71b-ceb972a66e25"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:25:12.199491 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:12.199428    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47668435-ed43-4f78-a71b-ceb972a66e25-kube-api-access-lztr9" (OuterVolumeSpecName: "kube-api-access-lztr9") pod "47668435-ed43-4f78-a71b-ceb972a66e25" (UID: "47668435-ed43-4f78-a71b-ceb972a66e25"). InnerVolumeSpecName "kube-api-access-lztr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:25:12.199610 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:12.199580    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47668435-ed43-4f78-a71b-ceb972a66e25-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "47668435-ed43-4f78-a71b-ceb972a66e25" (UID: "47668435-ed43-4f78-a71b-ceb972a66e25"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 14:25:12.201154 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:12.201133    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a77c87-7456v" event={"ID":"d4610e90-8580-4fcc-8c74-165ae57d2144","Type":"ContainerStarted","Data":"d5c055ba267f752b8193cb6a7b8302ab62325235ede07f42f9b3b9f524b424d1"}
Sep 17 14:25:12.201242 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:12.201177    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a77c87-7456v" event={"ID":"d4610e90-8580-4fcc-8c74-165ae57d2144","Type":"ContainerStarted","Data":"c2bd14171880da4130f536a92930019b084ca7df953eb1f9ad1934db5ec2a287"}
Sep 17 14:25:12.209305 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:12.209288    2560 scope.go:117] "RemoveContainer" containerID="cd85126c847fb7d5a7f45f2cc20ca0a4936ad72022acd9bca6c3598ed7c0d204"
Sep 17 14:25:12.217954 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:12.217934    2560 scope.go:117] "RemoveContainer" containerID="ad05498cd1fe227512c4c17430f86d321638ff7a9eda259aaedefed815f08517"
Sep 17 14:25:12.218203 ip-10-0-129-158 kubenswrapper[2560]: E0917 14:25:12.218186    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad05498cd1fe227512c4c17430f86d321638ff7a9eda259aaedefed815f08517\": container with ID starting with ad05498cd1fe227512c4c17430f86d321638ff7a9eda259aaedefed815f08517 not found: ID does not exist" containerID="ad05498cd1fe227512c4c17430f86d321638ff7a9eda259aaedefed815f08517"
Sep 17 14:25:12.218253 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:12.218210    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad05498cd1fe227512c4c17430f86d321638ff7a9eda259aaedefed815f08517"} err="failed to get container status \"ad05498cd1fe227512c4c17430f86d321638ff7a9eda259aaedefed815f08517\": rpc error: code = NotFound desc = could not find container \"ad05498cd1fe227512c4c17430f86d321638ff7a9eda259aaedefed815f08517\": container with ID starting with ad05498cd1fe227512c4c17430f86d321638ff7a9eda259aaedefed815f08517 not found: ID does not exist"
Sep 17 14:25:12.218253 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:12.218225    2560 scope.go:117] "RemoveContainer" containerID="cd85126c847fb7d5a7f45f2cc20ca0a4936ad72022acd9bca6c3598ed7c0d204"
Sep 17 14:25:12.218506 ip-10-0-129-158 kubenswrapper[2560]: E0917 14:25:12.218491    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd85126c847fb7d5a7f45f2cc20ca0a4936ad72022acd9bca6c3598ed7c0d204\": container with ID starting with cd85126c847fb7d5a7f45f2cc20ca0a4936ad72022acd9bca6c3598ed7c0d204 not found: ID does not exist" containerID="cd85126c847fb7d5a7f45f2cc20ca0a4936ad72022acd9bca6c3598ed7c0d204"
Sep 17 14:25:12.218567 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:12.218510    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd85126c847fb7d5a7f45f2cc20ca0a4936ad72022acd9bca6c3598ed7c0d204"} err="failed to get container status \"cd85126c847fb7d5a7f45f2cc20ca0a4936ad72022acd9bca6c3598ed7c0d204\": rpc error: code = NotFound desc = could not find container \"cd85126c847fb7d5a7f45f2cc20ca0a4936ad72022acd9bca6c3598ed7c0d204\": container with ID starting with cd85126c847fb7d5a7f45f2cc20ca0a4936ad72022acd9bca6c3598ed7c0d204 not found: ID does not exist"
Sep 17 14:25:12.228104 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:12.228068    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a77c87-7456v" podStartSLOduration=2.228056835 podStartE2EDuration="2.228056835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:25:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:25:12.227877988 +0000 UTC m=+1781.170021634" watchObservedRunningTime="2026-09-17 14:25:12.228056835 +0000 UTC m=+1781.170200494"
Sep 17 14:25:12.298531 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:12.298504    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lztr9\" (UniqueName: \"kubernetes.io/projected/47668435-ed43-4f78-a71b-ceb972a66e25-kube-api-access-lztr9\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:25:12.298607 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:12.298532    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47668435-ed43-4f78-a71b-ceb972a66e25-istio-podinfo\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:25:12.298607 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:12.298547    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47668435-ed43-4f78-a71b-ceb972a66e25-istio-token\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:25:12.298607 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:12.298561    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47668435-ed43-4f78-a71b-ceb972a66e25-istio-envoy\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:25:12.529816 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:12.529779    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s"]
Sep 17 14:25:12.536551 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:12.536523    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d9b17d-openshift-default-6f7786f4fd-cfr6s"]
Sep 17 14:25:13.209678 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:13.209645    2560 generic.go:356] "Generic (PLEG): container finished" podID="d4610e90-8580-4fcc-8c74-165ae57d2144" containerID="d5c055ba267f752b8193cb6a7b8302ab62325235ede07f42f9b3b9f524b424d1" exitCode=0
Sep 17 14:25:13.209877 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:13.209737    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a77c87-7456v" event={"ID":"d4610e90-8580-4fcc-8c74-165ae57d2144","Type":"ContainerDied","Data":"d5c055ba267f752b8193cb6a7b8302ab62325235ede07f42f9b3b9f524b424d1"}
Sep 17 14:25:13.217397 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:13.217374    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:13.570384 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:13.570354    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47668435-ed43-4f78-a71b-ceb972a66e25" path="/var/lib/kubelet/pods/47668435-ed43-4f78-a71b-ceb972a66e25/volumes"
Sep 17 14:25:14.195272 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:14.195241    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:14.360114 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:14.360093    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a77c87-7456v"
Sep 17 14:25:14.517868 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:14.517792    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4610e90-8580-4fcc-8c74-165ae57d2144-maas-api-service-ca\") pod \"d4610e90-8580-4fcc-8c74-165ae57d2144\" (UID: \"d4610e90-8580-4fcc-8c74-165ae57d2144\") "
Sep 17 14:25:14.518110 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:14.518090    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4610e90-8580-4fcc-8c74-165ae57d2144-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d4610e90-8580-4fcc-8c74-165ae57d2144" (UID: "d4610e90-8580-4fcc-8c74-165ae57d2144"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:25:14.618710 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:14.618685    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4610e90-8580-4fcc-8c74-165ae57d2144-maas-api-service-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:25:15.055423 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:15.055381    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:15.224720 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:15.224696    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a77c87-7456v"
Sep 17 14:25:15.224883 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:15.224731    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a77c87-7456v" event={"ID":"d4610e90-8580-4fcc-8c74-165ae57d2144","Type":"ContainerDied","Data":"c2bd14171880da4130f536a92930019b084ca7df953eb1f9ad1934db5ec2a287"}
Sep 17 14:25:15.224883 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:15.224775    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2bd14171880da4130f536a92930019b084ca7df953eb1f9ad1934db5ec2a287"
Sep 17 14:25:16.133735 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:16.133700    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:16.977147 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:16.977113    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:17.824508 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:17.824453    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:21.866350 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:21.866313    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:21.967138 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:21.967107    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:22.033573 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:22.033541    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:23.435122 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:23.435093    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:27.667809 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:27.667716    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:27.738212 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:27.738179    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:27.803743 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:27.803710    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:27.891568 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:27.891539    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a77c87-7456v"]
Sep 17 14:25:27.893436 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:27.893406    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:27.894721 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:27.894700    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a77c87-7456v"]
Sep 17 14:25:27.959339 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:27.959257    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:28.255143 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:28.255058    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:28.366913 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:28.366877    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:29.110866 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:29.110826    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:29.570496 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:29.570445    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4610e90-8580-4fcc-8c74-165ae57d2144" path="/var/lib/kubelet/pods/d4610e90-8580-4fcc-8c74-165ae57d2144/volumes"
Sep 17 14:25:30.053790 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:30.053761    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:30.875247 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:30.875212    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:31.934339 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:31.934310    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:31.970366 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:31.970338    2560 scope.go:117] "RemoveContainer" containerID="fc82eda55820ffdeace6372738ec079690091d4abfd57cefff79a7192d44667d"
Sep 17 14:25:31.979561 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:31.979542    2560 scope.go:117] "RemoveContainer" containerID="7c018fc05ddf363b4d5de88bba3bf168ba78ab9eb48226de81bdfb18c8cf996f"
Sep 17 14:25:32.742411 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:32.742374    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:34.908363 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:34.908323    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:35.009390 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:35.009361    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:35.156364 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:35.156335    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv"]
Sep 17 14:25:35.156595 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:35.156573    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv" podUID="f909bf57-b7b6-4e2b-b054-e06b7385a90a" containerName="istio-proxy" containerID="cri-o://50ea7864495aa34217865296bc4db68b5473ed5b503c5c346021f98ffc032e35" gracePeriod=30
Sep 17 14:25:35.169903 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:35.169833    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:35.258842 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:35.258806    2560 patch_prober.go:28] interesting pod/e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 17 14:25:35.258987 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:35.258878    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv" podUID="f909bf57-b7b6-4e2b-b054-e06b7385a90a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 14:25:35.571952 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:35.571925    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2b8275-tfgdq"]
Sep 17 14:25:35.572342 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:35.572328    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47668435-ed43-4f78-a71b-ceb972a66e25" containerName="istio-proxy"
Sep 17 14:25:35.572386 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:35.572343    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="47668435-ed43-4f78-a71b-ceb972a66e25" containerName="istio-proxy"
Sep 17 14:25:35.572386 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:35.572355    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47668435-ed43-4f78-a71b-ceb972a66e25" containerName="istio-proxy"
Sep 17 14:25:35.572386 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:35.572361    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="47668435-ed43-4f78-a71b-ceb972a66e25" containerName="istio-proxy"
Sep 17 14:25:35.572386 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:35.572369    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4610e90-8580-4fcc-8c74-165ae57d2144" containerName="revoke-keys"
Sep 17 14:25:35.572386 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:35.572375    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4610e90-8580-4fcc-8c74-165ae57d2144" containerName="revoke-keys"
Sep 17 14:25:35.572561 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:35.572447    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4610e90-8580-4fcc-8c74-165ae57d2144" containerName="revoke-keys"
Sep 17 14:25:35.572561 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:35.572469    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="47668435-ed43-4f78-a71b-ceb972a66e25" containerName="istio-proxy"
Sep 17 14:25:35.572561 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:35.572475    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="47668435-ed43-4f78-a71b-ceb972a66e25" containerName="istio-proxy"
Sep 17 14:25:35.575544 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:35.575529    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2b8275-tfgdq"
Sep 17 14:25:35.579174 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:35.579152    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 14:25:35.579303 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:35.579202    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rp9ww\""
Sep 17 14:25:35.593084 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:35.593059    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2b8275-tfgdq"]
Sep 17 14:25:35.621716 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:35.621689    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27940c87-d005-4ef1-bc3b-f46f219e38f0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-2b8275-tfgdq\" (UID: \"27940c87-d005-4ef1-bc3b-f46f219e38f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2b8275-tfgdq"
Sep 17 14:25:35.722358 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:35.722327    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27940c87-d005-4ef1-bc3b-f46f219e38f0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-2b8275-tfgdq\" (UID: \"27940c87-d005-4ef1-bc3b-f46f219e38f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2b8275-tfgdq"
Sep 17 14:25:35.722904 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:35.722886    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27940c87-d005-4ef1-bc3b-f46f219e38f0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-2b8275-tfgdq\" (UID: \"27940c87-d005-4ef1-bc3b-f46f219e38f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2b8275-tfgdq"
Sep 17 14:25:35.769423 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:35.769396    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:35.890557 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:35.890481    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2b8275-tfgdq"
Sep 17 14:25:36.160967 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:36.160942    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2b8275-tfgdq"]
Sep 17 14:25:36.162395 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:25:36.162370    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27940c87_d005_4ef1_bc3b_f46f219e38f0.slice/crio-1315d1faeff6acd0e0474d9616ebf629e61b4d36eb4bc9d665329f7d7b42a678 WatchSource:0}: Error finding container 1315d1faeff6acd0e0474d9616ebf629e61b4d36eb4bc9d665329f7d7b42a678: Status 404 returned error can't find the container with id 1315d1faeff6acd0e0474d9616ebf629e61b4d36eb4bc9d665329f7d7b42a678
Sep 17 14:25:36.364885 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:36.364856    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2b8275-tfgdq" event={"ID":"27940c87-d005-4ef1-bc3b-f46f219e38f0","Type":"ContainerStarted","Data":"1315d1faeff6acd0e0474d9616ebf629e61b4d36eb4bc9d665329f7d7b42a678"}
Sep 17 14:25:36.542067 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:36.542032    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:37.372902 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:37.372872    2560 generic.go:356] "Generic (PLEG): container finished" podID="27940c87-d005-4ef1-bc3b-f46f219e38f0" containerID="f66bac8bc10a559a4960182bf6d46b44572d1ec71e6049c2a858f0308eae8667" exitCode=0
Sep 17 14:25:37.373275 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:37.372962    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2b8275-tfgdq" event={"ID":"27940c87-d005-4ef1-bc3b-f46f219e38f0","Type":"ContainerDied","Data":"f66bac8bc10a559a4960182bf6d46b44572d1ec71e6049c2a858f0308eae8667"}
Sep 17 14:25:37.563669 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:37.563632    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:38.395804 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:38.395767    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:38.511007 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:38.510984    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2b8275-tfgdq"
Sep 17 14:25:38.551173 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:38.551141    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27940c87-d005-4ef1-bc3b-f46f219e38f0-maas-api-service-ca\") pod \"27940c87-d005-4ef1-bc3b-f46f219e38f0\" (UID: \"27940c87-d005-4ef1-bc3b-f46f219e38f0\") "
Sep 17 14:25:38.551389 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:38.551371    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27940c87-d005-4ef1-bc3b-f46f219e38f0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "27940c87-d005-4ef1-bc3b-f46f219e38f0" (UID: "27940c87-d005-4ef1-bc3b-f46f219e38f0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:25:38.551562 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:38.551547    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27940c87-d005-4ef1-bc3b-f46f219e38f0-maas-api-service-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:25:39.393827 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:39.393797    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2b8275-tfgdq"
Sep 17 14:25:39.394029 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:39.393830    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2b8275-tfgdq" event={"ID":"27940c87-d005-4ef1-bc3b-f46f219e38f0","Type":"ContainerDied","Data":"1315d1faeff6acd0e0474d9616ebf629e61b4d36eb4bc9d665329f7d7b42a678"}
Sep 17 14:25:39.394029 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:39.393912    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1315d1faeff6acd0e0474d9616ebf629e61b4d36eb4bc9d665329f7d7b42a678"
Sep 17 14:25:39.522765 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:39.522711    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:40.402829 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:40.402803    2560 generic.go:356] "Generic (PLEG): container finished" podID="f909bf57-b7b6-4e2b-b054-e06b7385a90a" containerID="50ea7864495aa34217865296bc4db68b5473ed5b503c5c346021f98ffc032e35" exitCode=0
Sep 17 14:25:40.402920 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:40.402872    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv" event={"ID":"f909bf57-b7b6-4e2b-b054-e06b7385a90a","Type":"ContainerDied","Data":"50ea7864495aa34217865296bc4db68b5473ed5b503c5c346021f98ffc032e35"}
Sep 17 14:25:40.402969 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:40.402925    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv" event={"ID":"f909bf57-b7b6-4e2b-b054-e06b7385a90a","Type":"ContainerDied","Data":"732c2996668dc9579885d0258ae4247e1f1096b8bb262fb60cdaf5a050e964fc"}
Sep 17 14:25:40.402969 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:40.402943    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="732c2996668dc9579885d0258ae4247e1f1096b8bb262fb60cdaf5a050e964fc"
Sep 17 14:25:40.402969 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:40.402958    2560 scope.go:117] "RemoveContainer" containerID="1575f29d6dd2f965f8c0aa0c391d03ea8d9cb24fa1ce73e6ef735a21ceb58cf4"
Sep 17 14:25:40.416054 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:40.416038    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv"
Sep 17 14:25:40.470395 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:40.470318    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f909bf57-b7b6-4e2b-b054-e06b7385a90a-istiod-ca-cert\") pod \"f909bf57-b7b6-4e2b-b054-e06b7385a90a\" (UID: \"f909bf57-b7b6-4e2b-b054-e06b7385a90a\") "
Sep 17 14:25:40.470540 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:40.470418    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f909bf57-b7b6-4e2b-b054-e06b7385a90a-istio-envoy\") pod \"f909bf57-b7b6-4e2b-b054-e06b7385a90a\" (UID: \"f909bf57-b7b6-4e2b-b054-e06b7385a90a\") "
Sep 17 14:25:40.470540 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:40.470440    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f909bf57-b7b6-4e2b-b054-e06b7385a90a-workload-certs\") pod \"f909bf57-b7b6-4e2b-b054-e06b7385a90a\" (UID: \"f909bf57-b7b6-4e2b-b054-e06b7385a90a\") "
Sep 17 14:25:40.470540 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:40.470475    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f909bf57-b7b6-4e2b-b054-e06b7385a90a-istio-podinfo\") pod \"f909bf57-b7b6-4e2b-b054-e06b7385a90a\" (UID: \"f909bf57-b7b6-4e2b-b054-e06b7385a90a\") "
Sep 17 14:25:40.470540 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:40.470509    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8d26w\" (UniqueName: \"kubernetes.io/projected/f909bf57-b7b6-4e2b-b054-e06b7385a90a-kube-api-access-8d26w\") pod \"f909bf57-b7b6-4e2b-b054-e06b7385a90a\" (UID: \"f909bf57-b7b6-4e2b-b054-e06b7385a90a\") "
Sep 17 14:25:40.470789 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:40.470575    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f909bf57-b7b6-4e2b-b054-e06b7385a90a-credential-socket\") pod \"f909bf57-b7b6-4e2b-b054-e06b7385a90a\" (UID: \"f909bf57-b7b6-4e2b-b054-e06b7385a90a\") "
Sep 17 14:25:40.470789 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:40.470620    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f909bf57-b7b6-4e2b-b054-e06b7385a90a-istio-data\") pod \"f909bf57-b7b6-4e2b-b054-e06b7385a90a\" (UID: \"f909bf57-b7b6-4e2b-b054-e06b7385a90a\") "
Sep 17 14:25:40.470789 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:40.470646    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f909bf57-b7b6-4e2b-b054-e06b7385a90a-workload-socket\") pod \"f909bf57-b7b6-4e2b-b054-e06b7385a90a\" (UID: \"f909bf57-b7b6-4e2b-b054-e06b7385a90a\") "
Sep 17 14:25:40.470789 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:40.470671    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f909bf57-b7b6-4e2b-b054-e06b7385a90a-istio-token\") pod \"f909bf57-b7b6-4e2b-b054-e06b7385a90a\" (UID: \"f909bf57-b7b6-4e2b-b054-e06b7385a90a\") "
Sep 17 14:25:40.470789 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:40.470778    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f909bf57-b7b6-4e2b-b054-e06b7385a90a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f909bf57-b7b6-4e2b-b054-e06b7385a90a" (UID: "f909bf57-b7b6-4e2b-b054-e06b7385a90a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:25:40.471069 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:40.470794    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f909bf57-b7b6-4e2b-b054-e06b7385a90a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f909bf57-b7b6-4e2b-b054-e06b7385a90a" (UID: "f909bf57-b7b6-4e2b-b054-e06b7385a90a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:25:40.471069 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:40.470896    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f909bf57-b7b6-4e2b-b054-e06b7385a90a-istio-data" (OuterVolumeSpecName: "istio-data") pod "f909bf57-b7b6-4e2b-b054-e06b7385a90a" (UID: "f909bf57-b7b6-4e2b-b054-e06b7385a90a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:25:40.471069 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:40.470980    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f909bf57-b7b6-4e2b-b054-e06b7385a90a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f909bf57-b7b6-4e2b-b054-e06b7385a90a" (UID: "f909bf57-b7b6-4e2b-b054-e06b7385a90a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:25:40.471069 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:40.470996    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f909bf57-b7b6-4e2b-b054-e06b7385a90a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f909bf57-b7b6-4e2b-b054-e06b7385a90a" (UID: "f909bf57-b7b6-4e2b-b054-e06b7385a90a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:25:40.471295 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:40.471136    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f909bf57-b7b6-4e2b-b054-e06b7385a90a-workload-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:25:40.471295 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:40.471149    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f909bf57-b7b6-4e2b-b054-e06b7385a90a-credential-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:25:40.471295 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:40.471167    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f909bf57-b7b6-4e2b-b054-e06b7385a90a-istio-data\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:25:40.471295 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:40.471186    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f909bf57-b7b6-4e2b-b054-e06b7385a90a-workload-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:25:40.471295 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:40.471200    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f909bf57-b7b6-4e2b-b054-e06b7385a90a-istiod-ca-cert\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:25:40.472943 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:40.472908    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f909bf57-b7b6-4e2b-b054-e06b7385a90a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f909bf57-b7b6-4e2b-b054-e06b7385a90a" (UID: "f909bf57-b7b6-4e2b-b054-e06b7385a90a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 14:25:40.473045 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:40.472974    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f909bf57-b7b6-4e2b-b054-e06b7385a90a-istio-token" (OuterVolumeSpecName: "istio-token") pod "f909bf57-b7b6-4e2b-b054-e06b7385a90a" (UID: "f909bf57-b7b6-4e2b-b054-e06b7385a90a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:25:40.473169 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:40.473148    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f909bf57-b7b6-4e2b-b054-e06b7385a90a-kube-api-access-8d26w" (OuterVolumeSpecName: "kube-api-access-8d26w") pod "f909bf57-b7b6-4e2b-b054-e06b7385a90a" (UID: "f909bf57-b7b6-4e2b-b054-e06b7385a90a"). InnerVolumeSpecName "kube-api-access-8d26w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:25:40.473231 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:40.473207    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f909bf57-b7b6-4e2b-b054-e06b7385a90a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f909bf57-b7b6-4e2b-b054-e06b7385a90a" (UID: "f909bf57-b7b6-4e2b-b054-e06b7385a90a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:25:40.474944 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:40.474918    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:40.572517 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:40.572488    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f909bf57-b7b6-4e2b-b054-e06b7385a90a-istio-token\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:25:40.572517 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:40.572512    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f909bf57-b7b6-4e2b-b054-e06b7385a90a-istio-envoy\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:25:40.572517 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:40.572521    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f909bf57-b7b6-4e2b-b054-e06b7385a90a-istio-podinfo\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:25:40.572939 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:40.572532    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d26w\" (UniqueName: \"kubernetes.io/projected/f909bf57-b7b6-4e2b-b054-e06b7385a90a-kube-api-access-8d26w\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:25:41.410318 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:41.410284    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv"
Sep 17 14:25:41.457425 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:41.457395    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv"]
Sep 17 14:25:41.460089 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:41.460068    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-a77c87-openshift-default-5f67cb465b-k2xpv"]
Sep 17 14:25:41.569627 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:41.569599    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f909bf57-b7b6-4e2b-b054-e06b7385a90a" path="/var/lib/kubelet/pods/f909bf57-b7b6-4e2b-b054-e06b7385a90a/volumes"
Sep 17 14:25:41.674094 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:41.674011    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:41.788026 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:41.787996    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:46.148573 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:46.148538    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:46.245397 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:46.245362    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:46.346171 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:46.346139    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:46.365123 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:46.365097    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2b8275-tfgdq"]
Sep 17 14:25:46.367645 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:46.367626    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2b8275-tfgdq"]
Sep 17 14:25:46.446712 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:46.446610    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:46.763594 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:46.763512    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:46.860919 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:46.860875    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:47.570057 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:47.570030    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27940c87-d005-4ef1-bc3b-f46f219e38f0" path="/var/lib/kubelet/pods/27940c87-d005-4ef1-bc3b-f46f219e38f0/volumes"
Sep 17 14:25:47.808956 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:47.808923    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:48.636378 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:48.636348    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:49.638986 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:49.638950    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:50.467817 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:50.467781    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:54.398339 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:54.398295    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:54.496510 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:54.496475    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:54.668618 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:54.668521    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l"]
Sep 17 14:25:54.668826 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:54.668799    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l" podUID="52c3f621-606c-42b4-99b2-15dccb0ba8cc" containerName="istio-proxy" containerID="cri-o://c557480e3e9fab689bf05e9b2ccfc2303bc816c5ae3167516e7de1ce243f9f8b" gracePeriod=30
Sep 17 14:25:54.670653 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:54.670630    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:54.995215 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:54.995126    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6e2bb9-lqhpj"]
Sep 17 14:25:54.995666 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:54.995648    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f909bf57-b7b6-4e2b-b054-e06b7385a90a" containerName="istio-proxy"
Sep 17 14:25:54.995666 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:54.995667    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f909bf57-b7b6-4e2b-b054-e06b7385a90a" containerName="istio-proxy"
Sep 17 14:25:54.995820 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:54.995712    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27940c87-d005-4ef1-bc3b-f46f219e38f0" containerName="revoke-keys"
Sep 17 14:25:54.995820 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:54.995722    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="27940c87-d005-4ef1-bc3b-f46f219e38f0" containerName="revoke-keys"
Sep 17 14:25:54.995820 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:54.995816    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f909bf57-b7b6-4e2b-b054-e06b7385a90a" containerName="istio-proxy"
Sep 17 14:25:54.995968 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:54.995830    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f909bf57-b7b6-4e2b-b054-e06b7385a90a" containerName="istio-proxy"
Sep 17 14:25:54.995968 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:54.995844    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="27940c87-d005-4ef1-bc3b-f46f219e38f0" containerName="revoke-keys"
Sep 17 14:25:55.000380 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:55.000359    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6e2bb9-lqhpj"
Sep 17 14:25:55.002708 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:55.002690    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 14:25:55.002953 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:55.002930    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rp9ww\""
Sep 17 14:25:55.007397 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:55.007378    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6e2bb9-lqhpj"]
Sep 17 14:25:55.109977 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:55.109942    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54cf304b-fe9c-44c5-b2b9-ddca85919c1a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6e2bb9-lqhpj\" (UID: \"54cf304b-fe9c-44c5-b2b9-ddca85919c1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6e2bb9-lqhpj"
Sep 17 14:25:55.210699 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:55.210660    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54cf304b-fe9c-44c5-b2b9-ddca85919c1a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6e2bb9-lqhpj\" (UID: \"54cf304b-fe9c-44c5-b2b9-ddca85919c1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6e2bb9-lqhpj"
Sep 17 14:25:55.211261 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:55.211242    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54cf304b-fe9c-44c5-b2b9-ddca85919c1a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6e2bb9-lqhpj\" (UID: \"54cf304b-fe9c-44c5-b2b9-ddca85919c1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6e2bb9-lqhpj"
Sep 17 14:25:55.311871 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:55.311836    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6e2bb9-lqhpj"
Sep 17 14:25:55.312059 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:55.311910    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:55.598001 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:55.597975    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6e2bb9-lqhpj"]
Sep 17 14:25:55.600856 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:25:55.600829    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54cf304b_fe9c_44c5_b2b9_ddca85919c1a.slice/crio-4275e3724c73e707522ba8af604b59c4dff9cf61a6766940b8e99ed4f3fc11a9 WatchSource:0}: Error finding container 4275e3724c73e707522ba8af604b59c4dff9cf61a6766940b8e99ed4f3fc11a9: Status 404 returned error can't find the container with id 4275e3724c73e707522ba8af604b59c4dff9cf61a6766940b8e99ed4f3fc11a9
Sep 17 14:25:56.508841 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:56.508743    2560 generic.go:356] "Generic (PLEG): container finished" podID="54cf304b-fe9c-44c5-b2b9-ddca85919c1a" containerID="b0fa5a624a11a8db3db07f630d8211ba0dc40f58ff2f622e979496811c7d95de" exitCode=0
Sep 17 14:25:56.508841 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:56.508791    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6e2bb9-lqhpj" event={"ID":"54cf304b-fe9c-44c5-b2b9-ddca85919c1a","Type":"ContainerDied","Data":"b0fa5a624a11a8db3db07f630d8211ba0dc40f58ff2f622e979496811c7d95de"}
Sep 17 14:25:56.509070 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:56.508875    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6e2bb9-lqhpj" event={"ID":"54cf304b-fe9c-44c5-b2b9-ddca85919c1a","Type":"ContainerStarted","Data":"4275e3724c73e707522ba8af604b59c4dff9cf61a6766940b8e99ed4f3fc11a9"}
Sep 17 14:25:56.679166 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:56.679133    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:57.652898 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:57.652875    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6e2bb9-lqhpj"
Sep 17 14:25:57.716564 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:57.716526    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:57.837453 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:57.837422    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54cf304b-fe9c-44c5-b2b9-ddca85919c1a-maas-api-service-ca\") pod \"54cf304b-fe9c-44c5-b2b9-ddca85919c1a\" (UID: \"54cf304b-fe9c-44c5-b2b9-ddca85919c1a\") "
Sep 17 14:25:57.837753 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:57.837731    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54cf304b-fe9c-44c5-b2b9-ddca85919c1a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "54cf304b-fe9c-44c5-b2b9-ddca85919c1a" (UID: "54cf304b-fe9c-44c5-b2b9-ddca85919c1a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:25:57.837901 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:57.837888    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54cf304b-fe9c-44c5-b2b9-ddca85919c1a-maas-api-service-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:25:58.440937 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:58.440902    2560 patch_prober.go:28] interesting pod/e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 17 14:25:58.441102 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:58.440954    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l" podUID="52c3f621-606c-42b4-99b2-15dccb0ba8cc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 14:25:58.522540 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:58.522513    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6e2bb9-lqhpj"
Sep 17 14:25:58.522540 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:58.522517    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6e2bb9-lqhpj" event={"ID":"54cf304b-fe9c-44c5-b2b9-ddca85919c1a","Type":"ContainerDied","Data":"4275e3724c73e707522ba8af604b59c4dff9cf61a6766940b8e99ed4f3fc11a9"}
Sep 17 14:25:58.522727 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:58.522562    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4275e3724c73e707522ba8af604b59c4dff9cf61a6766940b8e99ed4f3fc11a9"
Sep 17 14:25:58.613159 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:58.613130    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:59.466997 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:59.466951    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:59.915958 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:25:59.915937    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l"
Sep 17 14:26:00.060811 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:00.060780    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c3f621-606c-42b4-99b2-15dccb0ba8cc-workload-socket\") pod \"52c3f621-606c-42b4-99b2-15dccb0ba8cc\" (UID: \"52c3f621-606c-42b4-99b2-15dccb0ba8cc\") "
Sep 17 14:26:00.061001 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:00.060820    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52c3f621-606c-42b4-99b2-15dccb0ba8cc-istio-data\") pod \"52c3f621-606c-42b4-99b2-15dccb0ba8cc\" (UID: \"52c3f621-606c-42b4-99b2-15dccb0ba8cc\") "
Sep 17 14:26:00.061001 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:00.060854    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-swgzh\" (UniqueName: \"kubernetes.io/projected/52c3f621-606c-42b4-99b2-15dccb0ba8cc-kube-api-access-swgzh\") pod \"52c3f621-606c-42b4-99b2-15dccb0ba8cc\" (UID: \"52c3f621-606c-42b4-99b2-15dccb0ba8cc\") "
Sep 17 14:26:00.061001 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:00.060905    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52c3f621-606c-42b4-99b2-15dccb0ba8cc-istio-token\") pod \"52c3f621-606c-42b4-99b2-15dccb0ba8cc\" (UID: \"52c3f621-606c-42b4-99b2-15dccb0ba8cc\") "
Sep 17 14:26:00.061001 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:00.060934    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52c3f621-606c-42b4-99b2-15dccb0ba8cc-istio-envoy\") pod \"52c3f621-606c-42b4-99b2-15dccb0ba8cc\" (UID: \"52c3f621-606c-42b4-99b2-15dccb0ba8cc\") "
Sep 17 14:26:00.061001 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:00.060954    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52c3f621-606c-42b4-99b2-15dccb0ba8cc-workload-certs\") pod \"52c3f621-606c-42b4-99b2-15dccb0ba8cc\" (UID: \"52c3f621-606c-42b4-99b2-15dccb0ba8cc\") "
Sep 17 14:26:00.061267 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:00.061068    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52c3f621-606c-42b4-99b2-15dccb0ba8cc-istio-data" (OuterVolumeSpecName: "istio-data") pod "52c3f621-606c-42b4-99b2-15dccb0ba8cc" (UID: "52c3f621-606c-42b4-99b2-15dccb0ba8cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:26:00.061267 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:00.061086    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52c3f621-606c-42b4-99b2-15dccb0ba8cc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "52c3f621-606c-42b4-99b2-15dccb0ba8cc" (UID: "52c3f621-606c-42b4-99b2-15dccb0ba8cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:26:00.061267 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:00.061113    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52c3f621-606c-42b4-99b2-15dccb0ba8cc-istiod-ca-cert\") pod \"52c3f621-606c-42b4-99b2-15dccb0ba8cc\" (UID: \"52c3f621-606c-42b4-99b2-15dccb0ba8cc\") "
Sep 17 14:26:00.061267 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:00.061154    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52c3f621-606c-42b4-99b2-15dccb0ba8cc-istio-podinfo\") pod \"52c3f621-606c-42b4-99b2-15dccb0ba8cc\" (UID: \"52c3f621-606c-42b4-99b2-15dccb0ba8cc\") "
Sep 17 14:26:00.061267 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:00.061249    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52c3f621-606c-42b4-99b2-15dccb0ba8cc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "52c3f621-606c-42b4-99b2-15dccb0ba8cc" (UID: "52c3f621-606c-42b4-99b2-15dccb0ba8cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:26:00.061560 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:00.061272    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c3f621-606c-42b4-99b2-15dccb0ba8cc-credential-socket\") pod \"52c3f621-606c-42b4-99b2-15dccb0ba8cc\" (UID: \"52c3f621-606c-42b4-99b2-15dccb0ba8cc\") "
Sep 17 14:26:00.061560 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:00.061488    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52c3f621-606c-42b4-99b2-15dccb0ba8cc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "52c3f621-606c-42b4-99b2-15dccb0ba8cc" (UID: "52c3f621-606c-42b4-99b2-15dccb0ba8cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:26:00.061669 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:00.061600    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52c3f621-606c-42b4-99b2-15dccb0ba8cc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "52c3f621-606c-42b4-99b2-15dccb0ba8cc" (UID: "52c3f621-606c-42b4-99b2-15dccb0ba8cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:26:00.061815 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:00.061791    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52c3f621-606c-42b4-99b2-15dccb0ba8cc-workload-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:26:00.061815 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:00.061815    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52c3f621-606c-42b4-99b2-15dccb0ba8cc-istiod-ca-cert\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:26:00.062002 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:00.061824    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c3f621-606c-42b4-99b2-15dccb0ba8cc-credential-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:26:00.062002 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:00.061832    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c3f621-606c-42b4-99b2-15dccb0ba8cc-workload-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:26:00.062002 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:00.061841    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52c3f621-606c-42b4-99b2-15dccb0ba8cc-istio-data\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:26:00.063183 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:00.063159    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52c3f621-606c-42b4-99b2-15dccb0ba8cc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "52c3f621-606c-42b4-99b2-15dccb0ba8cc" (UID: "52c3f621-606c-42b4-99b2-15dccb0ba8cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 14:26:00.063366 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:00.063345    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52c3f621-606c-42b4-99b2-15dccb0ba8cc-istio-token" (OuterVolumeSpecName: "istio-token") pod "52c3f621-606c-42b4-99b2-15dccb0ba8cc" (UID: "52c3f621-606c-42b4-99b2-15dccb0ba8cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:26:00.063559 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:00.063536    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52c3f621-606c-42b4-99b2-15dccb0ba8cc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "52c3f621-606c-42b4-99b2-15dccb0ba8cc" (UID: "52c3f621-606c-42b4-99b2-15dccb0ba8cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:26:00.063618 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:00.063580    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52c3f621-606c-42b4-99b2-15dccb0ba8cc-kube-api-access-swgzh" (OuterVolumeSpecName: "kube-api-access-swgzh") pod "52c3f621-606c-42b4-99b2-15dccb0ba8cc" (UID: "52c3f621-606c-42b4-99b2-15dccb0ba8cc"). InnerVolumeSpecName "kube-api-access-swgzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:26:00.163280 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:00.163251    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swgzh\" (UniqueName: \"kubernetes.io/projected/52c3f621-606c-42b4-99b2-15dccb0ba8cc-kube-api-access-swgzh\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:26:00.163280 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:00.163277    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52c3f621-606c-42b4-99b2-15dccb0ba8cc-istio-token\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:26:00.163425 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:00.163291    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52c3f621-606c-42b4-99b2-15dccb0ba8cc-istio-envoy\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:26:00.163425 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:00.163302    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52c3f621-606c-42b4-99b2-15dccb0ba8cc-istio-podinfo\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:26:00.260475 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:00.260429    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:26:00.536366 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:00.536280    2560 generic.go:356] "Generic (PLEG): container finished" podID="52c3f621-606c-42b4-99b2-15dccb0ba8cc" containerID="c557480e3e9fab689bf05e9b2ccfc2303bc816c5ae3167516e7de1ce243f9f8b" exitCode=0
Sep 17 14:26:00.536366 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:00.536327    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l"
Sep 17 14:26:00.536366 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:00.536338    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l" event={"ID":"52c3f621-606c-42b4-99b2-15dccb0ba8cc","Type":"ContainerDied","Data":"c557480e3e9fab689bf05e9b2ccfc2303bc816c5ae3167516e7de1ce243f9f8b"}
Sep 17 14:26:00.536911 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:00.536392    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l" event={"ID":"52c3f621-606c-42b4-99b2-15dccb0ba8cc","Type":"ContainerDied","Data":"19ad9856825d027e52c654cb120a778ba5093bd540a09697633dbc6af5b8cd32"}
Sep 17 14:26:00.536911 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:00.536425    2560 scope.go:117] "RemoveContainer" containerID="c557480e3e9fab689bf05e9b2ccfc2303bc816c5ae3167516e7de1ce243f9f8b"
Sep 17 14:26:00.549777 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:00.549755    2560 scope.go:117] "RemoveContainer" containerID="e1c5e30ee457607e5c5f92e734776ddb9351f8d9c96432909da8d4577e46d19f"
Sep 17 14:26:00.558170 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:00.558154    2560 scope.go:117] "RemoveContainer" containerID="c557480e3e9fab689bf05e9b2ccfc2303bc816c5ae3167516e7de1ce243f9f8b"
Sep 17 14:26:00.558415 ip-10-0-129-158 kubenswrapper[2560]: E0917 14:26:00.558398    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c557480e3e9fab689bf05e9b2ccfc2303bc816c5ae3167516e7de1ce243f9f8b\": container with ID starting with c557480e3e9fab689bf05e9b2ccfc2303bc816c5ae3167516e7de1ce243f9f8b not found: ID does not exist" containerID="c557480e3e9fab689bf05e9b2ccfc2303bc816c5ae3167516e7de1ce243f9f8b"
Sep 17 14:26:00.558482 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:00.558423    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c557480e3e9fab689bf05e9b2ccfc2303bc816c5ae3167516e7de1ce243f9f8b"} err="failed to get container status \"c557480e3e9fab689bf05e9b2ccfc2303bc816c5ae3167516e7de1ce243f9f8b\": rpc error: code = NotFound desc = could not find container \"c557480e3e9fab689bf05e9b2ccfc2303bc816c5ae3167516e7de1ce243f9f8b\": container with ID starting with c557480e3e9fab689bf05e9b2ccfc2303bc816c5ae3167516e7de1ce243f9f8b not found: ID does not exist"
Sep 17 14:26:00.558482 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:00.558438    2560 scope.go:117] "RemoveContainer" containerID="e1c5e30ee457607e5c5f92e734776ddb9351f8d9c96432909da8d4577e46d19f"
Sep 17 14:26:00.558738 ip-10-0-129-158 kubenswrapper[2560]: E0917 14:26:00.558719    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1c5e30ee457607e5c5f92e734776ddb9351f8d9c96432909da8d4577e46d19f\": container with ID starting with e1c5e30ee457607e5c5f92e734776ddb9351f8d9c96432909da8d4577e46d19f not found: ID does not exist" containerID="e1c5e30ee457607e5c5f92e734776ddb9351f8d9c96432909da8d4577e46d19f"
Sep 17 14:26:00.558792 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:00.558743    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1c5e30ee457607e5c5f92e734776ddb9351f8d9c96432909da8d4577e46d19f"} err="failed to get container status \"e1c5e30ee457607e5c5f92e734776ddb9351f8d9c96432909da8d4577e46d19f\": rpc error: code = NotFound desc = could not find container \"e1c5e30ee457607e5c5f92e734776ddb9351f8d9c96432909da8d4577e46d19f\": container with ID starting with e1c5e30ee457607e5c5f92e734776ddb9351f8d9c96432909da8d4577e46d19f not found: ID does not exist"
Sep 17 14:26:00.571294 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:00.571273    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l"]
Sep 17 14:26:00.573910 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:00.573892    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-2b8275-openshift-default-58cd787d94-5lh7l"]
Sep 17 14:26:01.569359 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:01.569322    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52c3f621-606c-42b4-99b2-15dccb0ba8cc" path="/var/lib/kubelet/pods/52c3f621-606c-42b4-99b2-15dccb0ba8cc/volumes"
Sep 17 14:26:05.710901 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:05.710859    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:26:05.776809 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:05.776772    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:26:05.876652 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:05.876616    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:26:05.925546 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:05.925517    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6e2bb9-lqhpj"]
Sep 17 14:26:05.928095 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:05.928069    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6e2bb9-lqhpj"]
Sep 17 14:26:05.939137 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:05.939110    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:26:06.244772 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:06.244732    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:26:06.347204 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:06.347167    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:26:07.122091 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:07.122057    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:26:07.569416 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:07.569384    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54cf304b-fe9c-44c5-b2b9-ddca85919c1a" path="/var/lib/kubelet/pods/54cf304b-fe9c-44c5-b2b9-ddca85919c1a/volumes"
Sep 17 14:26:08.011294 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:08.011203    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:26:08.770515 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:08.770479    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:26:12.815209 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:12.815173    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:26:12.927434 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:12.927401    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-079b38-w76ck"]
Sep 17 14:26:12.927887 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:12.927872    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52c3f621-606c-42b4-99b2-15dccb0ba8cc" containerName="istio-proxy"
Sep 17 14:26:12.927931 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:12.927889    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="52c3f621-606c-42b4-99b2-15dccb0ba8cc" containerName="istio-proxy"
Sep 17 14:26:12.927931 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:12.927902    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54cf304b-fe9c-44c5-b2b9-ddca85919c1a" containerName="revoke-keys"
Sep 17 14:26:12.927931 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:12.927908    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="54cf304b-fe9c-44c5-b2b9-ddca85919c1a" containerName="revoke-keys"
Sep 17 14:26:12.927931 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:12.927917    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52c3f621-606c-42b4-99b2-15dccb0ba8cc" containerName="istio-proxy"
Sep 17 14:26:12.927931 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:12.927922    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="52c3f621-606c-42b4-99b2-15dccb0ba8cc" containerName="istio-proxy"
Sep 17 14:26:12.927931 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:12.927929    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f909bf57-b7b6-4e2b-b054-e06b7385a90a" containerName="istio-proxy"
Sep 17 14:26:12.928119 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:12.927934    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f909bf57-b7b6-4e2b-b054-e06b7385a90a" containerName="istio-proxy"
Sep 17 14:26:12.928119 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:12.927994    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="54cf304b-fe9c-44c5-b2b9-ddca85919c1a" containerName="revoke-keys"
Sep 17 14:26:12.928119 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:12.928003    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="52c3f621-606c-42b4-99b2-15dccb0ba8cc" containerName="istio-proxy"
Sep 17 14:26:12.928119 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:12.928011    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="52c3f621-606c-42b4-99b2-15dccb0ba8cc" containerName="istio-proxy"
Sep 17 14:26:12.931348 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:12.931332    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-079b38-w76ck"
Sep 17 14:26:12.933559 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:12.933535    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 14:26:12.933658 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:12.933600    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rp9ww\""
Sep 17 14:26:12.939220 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:12.939193    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-079b38-w76ck"]
Sep 17 14:26:12.979599 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:12.979570    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f668ee9a-53a2-4e01-81be-d395bdbab332-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-079b38-w76ck\" (UID: \"f668ee9a-53a2-4e01-81be-d395bdbab332\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-079b38-w76ck"
Sep 17 14:26:13.080601 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:13.080543    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f668ee9a-53a2-4e01-81be-d395bdbab332-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-079b38-w76ck\" (UID: \"f668ee9a-53a2-4e01-81be-d395bdbab332\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-079b38-w76ck"
Sep 17 14:26:13.081173 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:13.081156    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f668ee9a-53a2-4e01-81be-d395bdbab332-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-079b38-w76ck\" (UID: \"f668ee9a-53a2-4e01-81be-d395bdbab332\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-079b38-w76ck"
Sep 17 14:26:13.242397 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:13.242372    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-079b38-w76ck"
Sep 17 14:26:13.354515 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:13.354420    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:26:13.454535 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:13.454506    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:26:13.515995 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:13.515970    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-079b38-w76ck"]
Sep 17 14:26:13.516966 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:26:13.516940    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf668ee9a_53a2_4e01_81be_d395bdbab332.slice/crio-110ba84ff8fe90b3890bbf30379eb72077281db8fe8c40c977130c6913c37a7b WatchSource:0}: Error finding container 110ba84ff8fe90b3890bbf30379eb72077281db8fe8c40c977130c6913c37a7b: Status 404 returned error can't find the container with id 110ba84ff8fe90b3890bbf30379eb72077281db8fe8c40c977130c6913c37a7b
Sep 17 14:26:13.610407 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:13.610340    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt"]
Sep 17 14:26:13.610590 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:13.610567    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt" podUID="c66fc21d-4519-4293-bc1f-4e9ae8fe2564" containerName="istio-proxy" containerID="cri-o://921815787c1cbc3457545392ec720572c154159e9331f33a9ef283f5be16c858" gracePeriod=30
Sep 17 14:26:13.618155 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:13.618132    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-079b38-w76ck" event={"ID":"f668ee9a-53a2-4e01-81be-d395bdbab332","Type":"ContainerStarted","Data":"110ba84ff8fe90b3890bbf30379eb72077281db8fe8c40c977130c6913c37a7b"}
Sep 17 14:26:13.627276 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:13.627251    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:26:13.956318 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:13.956283    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:26:14.625919 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:14.625888    2560 generic.go:356] "Generic (PLEG): container finished" podID="f668ee9a-53a2-4e01-81be-d395bdbab332" containerID="bd5a80e72752ded4c659858250ac3731838b2e9f07f8e20218cda2d8f898e049" exitCode=0
Sep 17 14:26:14.626077 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:14.625950    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-079b38-w76ck" event={"ID":"f668ee9a-53a2-4e01-81be-d395bdbab332","Type":"ContainerDied","Data":"bd5a80e72752ded4c659858250ac3731838b2e9f07f8e20218cda2d8f898e049"}
Sep 17 14:26:14.722837 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:14.722757    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:26:15.561189 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:15.561152    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:26:15.763057 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:15.763036    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-079b38-w76ck"
Sep 17 14:26:15.805932 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:15.805897    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f668ee9a-53a2-4e01-81be-d395bdbab332-maas-api-service-ca\") pod \"f668ee9a-53a2-4e01-81be-d395bdbab332\" (UID: \"f668ee9a-53a2-4e01-81be-d395bdbab332\") "
Sep 17 14:26:15.806190 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:15.806169    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f668ee9a-53a2-4e01-81be-d395bdbab332-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f668ee9a-53a2-4e01-81be-d395bdbab332" (UID: "f668ee9a-53a2-4e01-81be-d395bdbab332"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:26:15.806265 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:15.806248    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f668ee9a-53a2-4e01-81be-d395bdbab332-maas-api-service-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:26:16.339115 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:16.339079    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:26:16.639868 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:16.639790    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-079b38-w76ck"
Sep 17 14:26:16.639868 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:16.639839    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-079b38-w76ck" event={"ID":"f668ee9a-53a2-4e01-81be-d395bdbab332","Type":"ContainerDied","Data":"110ba84ff8fe90b3890bbf30379eb72077281db8fe8c40c977130c6913c37a7b"}
Sep 17 14:26:16.640246 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:16.639886    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="110ba84ff8fe90b3890bbf30379eb72077281db8fe8c40c977130c6913c37a7b"
Sep 17 14:26:17.805861 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:17.805830    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-079b38/distinct-2-w4-079b38-kserve-5546bf445c-kv4gf"]
Sep 17 14:26:17.811391 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:17.811365    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-079b38/distinct-w4-079b38-kserve-7fd69fb686-5drgf"]
Sep 17 14:26:17.914614 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:17.914579    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:26:18.870700 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:18.870670    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt"
Sep 17 14:26:18.933595 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:18.933521    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l7mpg\" (UniqueName: \"kubernetes.io/projected/c66fc21d-4519-4293-bc1f-4e9ae8fe2564-kube-api-access-l7mpg\") pod \"c66fc21d-4519-4293-bc1f-4e9ae8fe2564\" (UID: \"c66fc21d-4519-4293-bc1f-4e9ae8fe2564\") "
Sep 17 14:26:18.933740 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:18.933648    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c66fc21d-4519-4293-bc1f-4e9ae8fe2564-istio-podinfo\") pod \"c66fc21d-4519-4293-bc1f-4e9ae8fe2564\" (UID: \"c66fc21d-4519-4293-bc1f-4e9ae8fe2564\") "
Sep 17 14:26:18.933740 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:18.933686    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c66fc21d-4519-4293-bc1f-4e9ae8fe2564-credential-socket\") pod \"c66fc21d-4519-4293-bc1f-4e9ae8fe2564\" (UID: \"c66fc21d-4519-4293-bc1f-4e9ae8fe2564\") "
Sep 17 14:26:18.933740 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:18.933728    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c66fc21d-4519-4293-bc1f-4e9ae8fe2564-workload-socket\") pod \"c66fc21d-4519-4293-bc1f-4e9ae8fe2564\" (UID: \"c66fc21d-4519-4293-bc1f-4e9ae8fe2564\") "
Sep 17 14:26:18.933924 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:18.933748    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c66fc21d-4519-4293-bc1f-4e9ae8fe2564-workload-certs\") pod \"c66fc21d-4519-4293-bc1f-4e9ae8fe2564\" (UID: \"c66fc21d-4519-4293-bc1f-4e9ae8fe2564\") "
Sep 17 14:26:18.933924 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:18.933763    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c66fc21d-4519-4293-bc1f-4e9ae8fe2564-istio-envoy\") pod \"c66fc21d-4519-4293-bc1f-4e9ae8fe2564\" (UID: \"c66fc21d-4519-4293-bc1f-4e9ae8fe2564\") "
Sep 17 14:26:18.933924 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:18.933804    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c66fc21d-4519-4293-bc1f-4e9ae8fe2564-istio-token\") pod \"c66fc21d-4519-4293-bc1f-4e9ae8fe2564\" (UID: \"c66fc21d-4519-4293-bc1f-4e9ae8fe2564\") "
Sep 17 14:26:18.933924 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:18.933856    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c66fc21d-4519-4293-bc1f-4e9ae8fe2564-istiod-ca-cert\") pod \"c66fc21d-4519-4293-bc1f-4e9ae8fe2564\" (UID: \"c66fc21d-4519-4293-bc1f-4e9ae8fe2564\") "
Sep 17 14:26:18.933924 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:18.933884    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c66fc21d-4519-4293-bc1f-4e9ae8fe2564-istio-data\") pod \"c66fc21d-4519-4293-bc1f-4e9ae8fe2564\" (UID: \"c66fc21d-4519-4293-bc1f-4e9ae8fe2564\") "
Sep 17 14:26:18.934156 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:18.934063    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c66fc21d-4519-4293-bc1f-4e9ae8fe2564-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c66fc21d-4519-4293-bc1f-4e9ae8fe2564" (UID: "c66fc21d-4519-4293-bc1f-4e9ae8fe2564"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:26:18.934156 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:18.934075    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c66fc21d-4519-4293-bc1f-4e9ae8fe2564-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c66fc21d-4519-4293-bc1f-4e9ae8fe2564" (UID: "c66fc21d-4519-4293-bc1f-4e9ae8fe2564"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:26:18.934308 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:18.934289    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c66fc21d-4519-4293-bc1f-4e9ae8fe2564-workload-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:26:18.934366 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:18.934315    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c66fc21d-4519-4293-bc1f-4e9ae8fe2564-workload-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:26:18.934366 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:18.934288    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c66fc21d-4519-4293-bc1f-4e9ae8fe2564-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c66fc21d-4519-4293-bc1f-4e9ae8fe2564" (UID: "c66fc21d-4519-4293-bc1f-4e9ae8fe2564"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:26:18.934366 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:18.934293    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c66fc21d-4519-4293-bc1f-4e9ae8fe2564-istio-data" (OuterVolumeSpecName: "istio-data") pod "c66fc21d-4519-4293-bc1f-4e9ae8fe2564" (UID: "c66fc21d-4519-4293-bc1f-4e9ae8fe2564"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:26:18.934557 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:18.934388    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c66fc21d-4519-4293-bc1f-4e9ae8fe2564-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c66fc21d-4519-4293-bc1f-4e9ae8fe2564" (UID: "c66fc21d-4519-4293-bc1f-4e9ae8fe2564"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:26:18.935809 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:18.935779    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c66fc21d-4519-4293-bc1f-4e9ae8fe2564-kube-api-access-l7mpg" (OuterVolumeSpecName: "kube-api-access-l7mpg") pod "c66fc21d-4519-4293-bc1f-4e9ae8fe2564" (UID: "c66fc21d-4519-4293-bc1f-4e9ae8fe2564"). InnerVolumeSpecName "kube-api-access-l7mpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:26:18.935929 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:18.935868    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c66fc21d-4519-4293-bc1f-4e9ae8fe2564-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c66fc21d-4519-4293-bc1f-4e9ae8fe2564" (UID: "c66fc21d-4519-4293-bc1f-4e9ae8fe2564"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 14:26:18.935998 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:18.935942    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c66fc21d-4519-4293-bc1f-4e9ae8fe2564-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c66fc21d-4519-4293-bc1f-4e9ae8fe2564" (UID: "c66fc21d-4519-4293-bc1f-4e9ae8fe2564"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:26:18.936209 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:18.936192    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c66fc21d-4519-4293-bc1f-4e9ae8fe2564-istio-token" (OuterVolumeSpecName: "istio-token") pod "c66fc21d-4519-4293-bc1f-4e9ae8fe2564" (UID: "c66fc21d-4519-4293-bc1f-4e9ae8fe2564"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:26:19.034981 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:19.034959    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7mpg\" (UniqueName: \"kubernetes.io/projected/c66fc21d-4519-4293-bc1f-4e9ae8fe2564-kube-api-access-l7mpg\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:26:19.034981 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:19.034982    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c66fc21d-4519-4293-bc1f-4e9ae8fe2564-istio-podinfo\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:26:19.035112 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:19.034991    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c66fc21d-4519-4293-bc1f-4e9ae8fe2564-credential-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:26:19.035112 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:19.035001    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c66fc21d-4519-4293-bc1f-4e9ae8fe2564-istio-envoy\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:26:19.035112 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:19.035009    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c66fc21d-4519-4293-bc1f-4e9ae8fe2564-istio-token\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:26:19.035112 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:19.035016    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c66fc21d-4519-4293-bc1f-4e9ae8fe2564-istiod-ca-cert\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:26:19.035112 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:19.035023    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c66fc21d-4519-4293-bc1f-4e9ae8fe2564-istio-data\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:26:19.659682 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:19.659650    2560 generic.go:356] "Generic (PLEG): container finished" podID="c66fc21d-4519-4293-bc1f-4e9ae8fe2564" containerID="921815787c1cbc3457545392ec720572c154159e9331f33a9ef283f5be16c858" exitCode=0
Sep 17 14:26:19.659861 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:19.659707    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt" event={"ID":"c66fc21d-4519-4293-bc1f-4e9ae8fe2564","Type":"ContainerDied","Data":"921815787c1cbc3457545392ec720572c154159e9331f33a9ef283f5be16c858"}
Sep 17 14:26:19.659861 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:19.659709    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt"
Sep 17 14:26:19.659861 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:19.659748    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt" event={"ID":"c66fc21d-4519-4293-bc1f-4e9ae8fe2564","Type":"ContainerDied","Data":"b790898beb46d08725d4e68fc070a59b4c8233882d9a5b450a4c42a99d8359e7"}
Sep 17 14:26:19.659861 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:19.659770    2560 scope.go:117] "RemoveContainer" containerID="921815787c1cbc3457545392ec720572c154159e9331f33a9ef283f5be16c858"
Sep 17 14:26:19.671135 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:19.671046    2560 scope.go:117] "RemoveContainer" containerID="9880e4126cffa56d0bbb6f0975de61ee8898a364be70441398c8c624ce64a287"
Sep 17 14:26:19.677161 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:19.677140    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt"]
Sep 17 14:26:19.681382 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:19.681356    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6e2bb9-openshift-default-64656c457b-lsbqt"]
Sep 17 14:26:19.681745 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:19.681723    2560 scope.go:117] "RemoveContainer" containerID="921815787c1cbc3457545392ec720572c154159e9331f33a9ef283f5be16c858"
Sep 17 14:26:19.682008 ip-10-0-129-158 kubenswrapper[2560]: E0917 14:26:19.681988    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"921815787c1cbc3457545392ec720572c154159e9331f33a9ef283f5be16c858\": container with ID starting with 921815787c1cbc3457545392ec720572c154159e9331f33a9ef283f5be16c858 not found: ID does not exist" containerID="921815787c1cbc3457545392ec720572c154159e9331f33a9ef283f5be16c858"
Sep 17 14:26:19.682059 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:19.682015    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"921815787c1cbc3457545392ec720572c154159e9331f33a9ef283f5be16c858"} err="failed to get container status \"921815787c1cbc3457545392ec720572c154159e9331f33a9ef283f5be16c858\": rpc error: code = NotFound desc = could not find container \"921815787c1cbc3457545392ec720572c154159e9331f33a9ef283f5be16c858\": container with ID starting with 921815787c1cbc3457545392ec720572c154159e9331f33a9ef283f5be16c858 not found: ID does not exist"
Sep 17 14:26:19.682059 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:19.682035    2560 scope.go:117] "RemoveContainer" containerID="9880e4126cffa56d0bbb6f0975de61ee8898a364be70441398c8c624ce64a287"
Sep 17 14:26:19.682276 ip-10-0-129-158 kubenswrapper[2560]: E0917 14:26:19.682261    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9880e4126cffa56d0bbb6f0975de61ee8898a364be70441398c8c624ce64a287\": container with ID starting with 9880e4126cffa56d0bbb6f0975de61ee8898a364be70441398c8c624ce64a287 not found: ID does not exist" containerID="9880e4126cffa56d0bbb6f0975de61ee8898a364be70441398c8c624ce64a287"
Sep 17 14:26:19.682317 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:19.682279    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9880e4126cffa56d0bbb6f0975de61ee8898a364be70441398c8c624ce64a287"} err="failed to get container status \"9880e4126cffa56d0bbb6f0975de61ee8898a364be70441398c8c624ce64a287\": rpc error: code = NotFound desc = could not find container \"9880e4126cffa56d0bbb6f0975de61ee8898a364be70441398c8c624ce64a287\": container with ID starting with 9880e4126cffa56d0bbb6f0975de61ee8898a364be70441398c8c624ce64a287 not found: ID does not exist"
Sep 17 14:26:21.576758 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:21.576717    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c66fc21d-4519-4293-bc1f-4e9ae8fe2564" path="/var/lib/kubelet/pods/c66fc21d-4519-4293-bc1f-4e9ae8fe2564/volumes"
Sep 17 14:26:29.095405 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:29.095367    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:26:29.112043 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:29.112009    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-079b38-29827575-gp6bp"]
Sep 17 14:26:29.113672 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:29.113635    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-079b38-29827575-gp6bp"]
Sep 17 14:26:29.185788 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:29.185730    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:26:29.286112 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:29.286073    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:26:29.336856 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:29.336830    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-079b38-w76ck"]
Sep 17 14:26:29.342086 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:29.342062    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-079b38-w76ck"]
Sep 17 14:26:29.386718 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:29.386660    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:26:29.569905 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:29.569873    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f668ee9a-53a2-4e01-81be-d395bdbab332" path="/var/lib/kubelet/pods/f668ee9a-53a2-4e01-81be-d395bdbab332/volumes"
Sep 17 14:26:29.570158 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:29.570145    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f79912d1-de41-45e4-89b6-0db0e97c50c9" path="/var/lib/kubelet/pods/f79912d1-de41-45e4-89b6-0db0e97c50c9/volumes"
Sep 17 14:26:29.707426 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:29.707346    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:26:29.812097 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:29.812051    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:26:30.777232 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:30.777198    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:26:32.000378 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:32.000333    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:26:32.038869 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:32.038842    2560 scope.go:117] "RemoveContainer" containerID="d1a9bba4bbd6ef0fdfa136e61e7cd56764edd5da30c7bc56359cec64d4fb1279"
Sep 17 14:26:32.825116 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:32.825078    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-079b38/distinct-2-w4-079b38-kserve-5546bf445c-kv4gf" podUID="8bfb040f-2e96-4649-a866-73f7c960c31d" containerName="main" containerID="cri-o://9beb2b8e48da358394aae4f2a8dcd56422c7680e26b345642b4efca85e121626" gracePeriod=45
Sep 17 14:26:32.830131 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:32.830101    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-079b38/distinct-w4-079b38-kserve-7fd69fb686-5drgf" podUID="0cd1d4fd-bb83-4910-89b7-191b012d9268" containerName="main" containerID="cri-o://c471354c7271d61d050f2f00da3c6da7795d1ff59a7c267ff0d8cd8b8efb90e7" gracePeriod=45
Sep 17 14:26:33.091040 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:33.091017    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-079b38/distinct-2-w4-079b38-kserve-5546bf445c-kv4gf"
Sep 17 14:26:33.093953 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:33.093933    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-079b38/distinct-w4-079b38-kserve-7fd69fb686-5drgf"
Sep 17 14:26:33.274709 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:33.274672    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kxsmz\" (UniqueName: \"kubernetes.io/projected/0cd1d4fd-bb83-4910-89b7-191b012d9268-kube-api-access-kxsmz\") pod \"0cd1d4fd-bb83-4910-89b7-191b012d9268\" (UID: \"0cd1d4fd-bb83-4910-89b7-191b012d9268\") "
Sep 17 14:26:33.274709 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:33.274715    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bfb040f-2e96-4649-a866-73f7c960c31d-tls-certs\") pod \"8bfb040f-2e96-4649-a866-73f7c960c31d\" (UID: \"8bfb040f-2e96-4649-a866-73f7c960c31d\") "
Sep 17 14:26:33.274950 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:33.274754    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cd1d4fd-bb83-4910-89b7-191b012d9268-tmp-dir\") pod \"0cd1d4fd-bb83-4910-89b7-191b012d9268\" (UID: \"0cd1d4fd-bb83-4910-89b7-191b012d9268\") "
Sep 17 14:26:33.274950 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:33.274811    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bfb040f-2e96-4649-a866-73f7c960c31d-model-cache\") pod \"8bfb040f-2e96-4649-a866-73f7c960c31d\" (UID: \"8bfb040f-2e96-4649-a866-73f7c960c31d\") "
Sep 17 14:26:33.274950 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:33.274851    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bfb040f-2e96-4649-a866-73f7c960c31d-tmp-dir\") pod \"8bfb040f-2e96-4649-a866-73f7c960c31d\" (UID: \"8bfb040f-2e96-4649-a866-73f7c960c31d\") "
Sep 17 14:26:33.274950 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:33.274872    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bfb040f-2e96-4649-a866-73f7c960c31d-dshm\") pod \"8bfb040f-2e96-4649-a866-73f7c960c31d\" (UID: \"8bfb040f-2e96-4649-a866-73f7c960c31d\") "
Sep 17 14:26:33.274950 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:33.274913    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cd1d4fd-bb83-4910-89b7-191b012d9268-tls-certs\") pod \"0cd1d4fd-bb83-4910-89b7-191b012d9268\" (UID: \"0cd1d4fd-bb83-4910-89b7-191b012d9268\") "
Sep 17 14:26:33.274950 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:33.274950    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cd1d4fd-bb83-4910-89b7-191b012d9268-home\") pod \"0cd1d4fd-bb83-4910-89b7-191b012d9268\" (UID: \"0cd1d4fd-bb83-4910-89b7-191b012d9268\") "
Sep 17 14:26:33.275285 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:33.275002    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9h9b9\" (UniqueName: \"kubernetes.io/projected/8bfb040f-2e96-4649-a866-73f7c960c31d-kube-api-access-9h9b9\") pod \"8bfb040f-2e96-4649-a866-73f7c960c31d\" (UID: \"8bfb040f-2e96-4649-a866-73f7c960c31d\") "
Sep 17 14:26:33.275285 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:33.275024    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cd1d4fd-bb83-4910-89b7-191b012d9268-model-cache\") pod \"0cd1d4fd-bb83-4910-89b7-191b012d9268\" (UID: \"0cd1d4fd-bb83-4910-89b7-191b012d9268\") "
Sep 17 14:26:33.275285 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:33.275027    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cd1d4fd-bb83-4910-89b7-191b012d9268-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0cd1d4fd-bb83-4910-89b7-191b012d9268" (UID: "0cd1d4fd-bb83-4910-89b7-191b012d9268"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:26:33.275285 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:33.275030    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bfb040f-2e96-4649-a866-73f7c960c31d-model-cache" (OuterVolumeSpecName: "model-cache") pod "8bfb040f-2e96-4649-a866-73f7c960c31d" (UID: "8bfb040f-2e96-4649-a866-73f7c960c31d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:26:33.275285 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:33.275069    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cd1d4fd-bb83-4910-89b7-191b012d9268-dshm\") pod \"0cd1d4fd-bb83-4910-89b7-191b012d9268\" (UID: \"0cd1d4fd-bb83-4910-89b7-191b012d9268\") "
Sep 17 14:26:33.275285 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:33.275108    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bfb040f-2e96-4649-a866-73f7c960c31d-home\") pod \"8bfb040f-2e96-4649-a866-73f7c960c31d\" (UID: \"8bfb040f-2e96-4649-a866-73f7c960c31d\") "
Sep 17 14:26:33.275285 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:33.275145    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bfb040f-2e96-4649-a866-73f7c960c31d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8bfb040f-2e96-4649-a866-73f7c960c31d" (UID: "8bfb040f-2e96-4649-a866-73f7c960c31d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:26:33.275285 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:33.275178    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cd1d4fd-bb83-4910-89b7-191b012d9268-home" (OuterVolumeSpecName: "home") pod "0cd1d4fd-bb83-4910-89b7-191b012d9268" (UID: "0cd1d4fd-bb83-4910-89b7-191b012d9268"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:26:33.275736 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:33.275354    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cd1d4fd-bb83-4910-89b7-191b012d9268-model-cache" (OuterVolumeSpecName: "model-cache") pod "0cd1d4fd-bb83-4910-89b7-191b012d9268" (UID: "0cd1d4fd-bb83-4910-89b7-191b012d9268"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:26:33.275736 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:33.275499    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bfb040f-2e96-4649-a866-73f7c960c31d-model-cache\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:26:33.275736 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:33.275519    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bfb040f-2e96-4649-a866-73f7c960c31d-tmp-dir\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:26:33.275736 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:33.275531    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cd1d4fd-bb83-4910-89b7-191b012d9268-home\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:26:33.275736 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:33.275543    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cd1d4fd-bb83-4910-89b7-191b012d9268-model-cache\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:26:33.275736 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:33.275554    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cd1d4fd-bb83-4910-89b7-191b012d9268-tmp-dir\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:26:33.275736 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:33.275588    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bfb040f-2e96-4649-a866-73f7c960c31d-home" (OuterVolumeSpecName: "home") pod "8bfb040f-2e96-4649-a866-73f7c960c31d" (UID: "8bfb040f-2e96-4649-a866-73f7c960c31d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:26:33.277147 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:33.277121    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bfb040f-2e96-4649-a866-73f7c960c31d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8bfb040f-2e96-4649-a866-73f7c960c31d" (UID: "8bfb040f-2e96-4649-a866-73f7c960c31d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:26:33.277247 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:33.277189    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cd1d4fd-bb83-4910-89b7-191b012d9268-dshm" (OuterVolumeSpecName: "dshm") pod "0cd1d4fd-bb83-4910-89b7-191b012d9268" (UID: "0cd1d4fd-bb83-4910-89b7-191b012d9268"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:26:33.277373 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:33.277354    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cd1d4fd-bb83-4910-89b7-191b012d9268-kube-api-access-kxsmz" (OuterVolumeSpecName: "kube-api-access-kxsmz") pod "0cd1d4fd-bb83-4910-89b7-191b012d9268" (UID: "0cd1d4fd-bb83-4910-89b7-191b012d9268"). InnerVolumeSpecName "kube-api-access-kxsmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:26:33.277428 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:33.277386    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bfb040f-2e96-4649-a866-73f7c960c31d-kube-api-access-9h9b9" (OuterVolumeSpecName: "kube-api-access-9h9b9") pod "8bfb040f-2e96-4649-a866-73f7c960c31d" (UID: "8bfb040f-2e96-4649-a866-73f7c960c31d"). InnerVolumeSpecName "kube-api-access-9h9b9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:26:33.277593 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:33.277567    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bfb040f-2e96-4649-a866-73f7c960c31d-dshm" (OuterVolumeSpecName: "dshm") pod "8bfb040f-2e96-4649-a866-73f7c960c31d" (UID: "8bfb040f-2e96-4649-a866-73f7c960c31d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:26:33.277593 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:33.277586    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cd1d4fd-bb83-4910-89b7-191b012d9268-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0cd1d4fd-bb83-4910-89b7-191b012d9268" (UID: "0cd1d4fd-bb83-4910-89b7-191b012d9268"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:26:33.376250 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:33.376186    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cd1d4fd-bb83-4910-89b7-191b012d9268-dshm\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:26:33.376250 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:33.376207    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bfb040f-2e96-4649-a866-73f7c960c31d-home\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:26:33.376250 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:33.376216    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxsmz\" (UniqueName: \"kubernetes.io/projected/0cd1d4fd-bb83-4910-89b7-191b012d9268-kube-api-access-kxsmz\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:26:33.376250 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:33.376225    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bfb040f-2e96-4649-a866-73f7c960c31d-tls-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:26:33.376250 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:33.376233    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bfb040f-2e96-4649-a866-73f7c960c31d-dshm\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:26:33.376250 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:33.376240    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cd1d4fd-bb83-4910-89b7-191b012d9268-tls-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:26:33.376250 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:33.376248    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9h9b9\" (UniqueName: \"kubernetes.io/projected/8bfb040f-2e96-4649-a866-73f7c960c31d-kube-api-access-9h9b9\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:26:33.757697 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:33.757606    2560 generic.go:356] "Generic (PLEG): container finished" podID="8bfb040f-2e96-4649-a866-73f7c960c31d" containerID="9beb2b8e48da358394aae4f2a8dcd56422c7680e26b345642b4efca85e121626" exitCode=0
Sep 17 14:26:33.757697 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:33.757664    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-079b38/distinct-2-w4-079b38-kserve-5546bf445c-kv4gf"
Sep 17 14:26:33.757920 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:33.757690    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-079b38/distinct-2-w4-079b38-kserve-5546bf445c-kv4gf" event={"ID":"8bfb040f-2e96-4649-a866-73f7c960c31d","Type":"ContainerDied","Data":"9beb2b8e48da358394aae4f2a8dcd56422c7680e26b345642b4efca85e121626"}
Sep 17 14:26:33.757920 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:33.757737    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-079b38/distinct-2-w4-079b38-kserve-5546bf445c-kv4gf" event={"ID":"8bfb040f-2e96-4649-a866-73f7c960c31d","Type":"ContainerDied","Data":"ba0748217f15e4b5abd4475d91c5f8dc3d9a165130d564530fc7900ee8444db6"}
Sep 17 14:26:33.757920 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:33.757760    2560 scope.go:117] "RemoveContainer" containerID="9beb2b8e48da358394aae4f2a8dcd56422c7680e26b345642b4efca85e121626"
Sep 17 14:26:33.759302 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:33.759282    2560 generic.go:356] "Generic (PLEG): container finished" podID="0cd1d4fd-bb83-4910-89b7-191b012d9268" containerID="c471354c7271d61d050f2f00da3c6da7795d1ff59a7c267ff0d8cd8b8efb90e7" exitCode=0
Sep 17 14:26:33.759397 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:33.759318    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-079b38/distinct-w4-079b38-kserve-7fd69fb686-5drgf" event={"ID":"0cd1d4fd-bb83-4910-89b7-191b012d9268","Type":"ContainerDied","Data":"c471354c7271d61d050f2f00da3c6da7795d1ff59a7c267ff0d8cd8b8efb90e7"}
Sep 17 14:26:33.759397 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:33.759327    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-079b38/distinct-w4-079b38-kserve-7fd69fb686-5drgf"
Sep 17 14:26:33.759397 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:33.759346    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-079b38/distinct-w4-079b38-kserve-7fd69fb686-5drgf" event={"ID":"0cd1d4fd-bb83-4910-89b7-191b012d9268","Type":"ContainerDied","Data":"7b54f2bc78b1a2a26782a76df7f616d2775111c24f73b3d8d3b640edd7c76b48"}
Sep 17 14:26:33.767139 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:33.767120    2560 scope.go:117] "RemoveContainer" containerID="9beb2b8e48da358394aae4f2a8dcd56422c7680e26b345642b4efca85e121626"
Sep 17 14:26:33.767408 ip-10-0-129-158 kubenswrapper[2560]: E0917 14:26:33.767388    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9beb2b8e48da358394aae4f2a8dcd56422c7680e26b345642b4efca85e121626\": container with ID starting with 9beb2b8e48da358394aae4f2a8dcd56422c7680e26b345642b4efca85e121626 not found: ID does not exist" containerID="9beb2b8e48da358394aae4f2a8dcd56422c7680e26b345642b4efca85e121626"
Sep 17 14:26:33.767451 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:33.767415    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9beb2b8e48da358394aae4f2a8dcd56422c7680e26b345642b4efca85e121626"} err="failed to get container status \"9beb2b8e48da358394aae4f2a8dcd56422c7680e26b345642b4efca85e121626\": rpc error: code = NotFound desc = could not find container \"9beb2b8e48da358394aae4f2a8dcd56422c7680e26b345642b4efca85e121626\": container with ID starting with 9beb2b8e48da358394aae4f2a8dcd56422c7680e26b345642b4efca85e121626 not found: ID does not exist"
Sep 17 14:26:33.767451 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:33.767431    2560 scope.go:117] "RemoveContainer" containerID="c471354c7271d61d050f2f00da3c6da7795d1ff59a7c267ff0d8cd8b8efb90e7"
Sep 17 14:26:33.775553 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:33.775538    2560 scope.go:117] "RemoveContainer" containerID="c471354c7271d61d050f2f00da3c6da7795d1ff59a7c267ff0d8cd8b8efb90e7"
Sep 17 14:26:33.775784 ip-10-0-129-158 kubenswrapper[2560]: E0917 14:26:33.775768    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c471354c7271d61d050f2f00da3c6da7795d1ff59a7c267ff0d8cd8b8efb90e7\": container with ID starting with c471354c7271d61d050f2f00da3c6da7795d1ff59a7c267ff0d8cd8b8efb90e7 not found: ID does not exist" containerID="c471354c7271d61d050f2f00da3c6da7795d1ff59a7c267ff0d8cd8b8efb90e7"
Sep 17 14:26:33.775839 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:33.775789    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c471354c7271d61d050f2f00da3c6da7795d1ff59a7c267ff0d8cd8b8efb90e7"} err="failed to get container status \"c471354c7271d61d050f2f00da3c6da7795d1ff59a7c267ff0d8cd8b8efb90e7\": rpc error: code = NotFound desc = could not find container \"c471354c7271d61d050f2f00da3c6da7795d1ff59a7c267ff0d8cd8b8efb90e7\": container with ID starting with c471354c7271d61d050f2f00da3c6da7795d1ff59a7c267ff0d8cd8b8efb90e7 not found: ID does not exist"
Sep 17 14:26:33.782558 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:33.782536    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-079b38/distinct-2-w4-079b38-kserve-5546bf445c-kv4gf"]
Sep 17 14:26:33.786894 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:33.786874    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-079b38/distinct-2-w4-079b38-kserve-5546bf445c-kv4gf"]
Sep 17 14:26:33.803581 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:33.803560    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-079b38/distinct-w4-079b38-kserve-7fd69fb686-5drgf"]
Sep 17 14:26:33.805751 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:33.805728    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-079b38/distinct-w4-079b38-kserve-7fd69fb686-5drgf"]
Sep 17 14:26:35.570017 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:35.569985    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0cd1d4fd-bb83-4910-89b7-191b012d9268" path="/var/lib/kubelet/pods/0cd1d4fd-bb83-4910-89b7-191b012d9268/volumes"
Sep 17 14:26:35.570388 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:35.570375    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bfb040f-2e96-4649-a866-73f7c960c31d" path="/var/lib/kubelet/pods/8bfb040f-2e96-4649-a866-73f7c960c31d/volumes"
Sep 17 14:26:36.999021 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:36.998975    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:26:37.090181 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:37.090133    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:26:37.333226 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:37.333190    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:26:44.263646 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:44.263611    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7777-d1d3e3"]
Sep 17 14:26:44.264050 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:44.264007    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c66fc21d-4519-4293-bc1f-4e9ae8fe2564" containerName="istio-proxy"
Sep 17 14:26:44.264050 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:44.264019    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c66fc21d-4519-4293-bc1f-4e9ae8fe2564" containerName="istio-proxy"
Sep 17 14:26:44.264050 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:44.264027    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cd1d4fd-bb83-4910-89b7-191b012d9268" containerName="main"
Sep 17 14:26:44.264050 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:44.264034    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cd1d4fd-bb83-4910-89b7-191b012d9268" containerName="main"
Sep 17 14:26:44.264050 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:44.264048    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c66fc21d-4519-4293-bc1f-4e9ae8fe2564" containerName="istio-proxy"
Sep 17 14:26:44.264050 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:44.264053    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c66fc21d-4519-4293-bc1f-4e9ae8fe2564" containerName="istio-proxy"
Sep 17 14:26:44.264258 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:44.264060    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bfb040f-2e96-4649-a866-73f7c960c31d" containerName="main"
Sep 17 14:26:44.264258 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:44.264065    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bfb040f-2e96-4649-a866-73f7c960c31d" containerName="main"
Sep 17 14:26:44.264258 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:44.264091    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f668ee9a-53a2-4e01-81be-d395bdbab332" containerName="revoke-keys"
Sep 17 14:26:44.264258 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:44.264097    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f668ee9a-53a2-4e01-81be-d395bdbab332" containerName="revoke-keys"
Sep 17 14:26:44.264258 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:44.264154    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="c66fc21d-4519-4293-bc1f-4e9ae8fe2564" containerName="istio-proxy"
Sep 17 14:26:44.264258 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:44.264161    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cd1d4fd-bb83-4910-89b7-191b012d9268" containerName="main"
Sep 17 14:26:44.264258 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:44.264168    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f668ee9a-53a2-4e01-81be-d395bdbab332" containerName="revoke-keys"
Sep 17 14:26:44.264258 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:44.264175    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bfb040f-2e96-4649-a866-73f7c960c31d" containerName="main"
Sep 17 14:26:44.267593 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:44.267575    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7777-d1d3e3"
Sep 17 14:26:44.269767 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:44.269744    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-f8z2f\""
Sep 17 14:26:44.273606 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:44.273583    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7777-d1d3e3"]
Sep 17 14:26:44.369114 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:44.369086    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqh2w\" (UniqueName: \"kubernetes.io/projected/8538c3b5-c914-49af-a938-e694ae1a4842-kube-api-access-pqh2w\") pod \"test-curl-7777-d1d3e3\" (UID: \"8538c3b5-c914-49af-a938-e694ae1a4842\") " pod="opendatahub/test-curl-7777-d1d3e3"
Sep 17 14:26:44.469813 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:44.469788    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqh2w\" (UniqueName: \"kubernetes.io/projected/8538c3b5-c914-49af-a938-e694ae1a4842-kube-api-access-pqh2w\") pod \"test-curl-7777-d1d3e3\" (UID: \"8538c3b5-c914-49af-a938-e694ae1a4842\") " pod="opendatahub/test-curl-7777-d1d3e3"
Sep 17 14:26:44.477837 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:44.477810    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqh2w\" (UniqueName: \"kubernetes.io/projected/8538c3b5-c914-49af-a938-e694ae1a4842-kube-api-access-pqh2w\") pod \"test-curl-7777-d1d3e3\" (UID: \"8538c3b5-c914-49af-a938-e694ae1a4842\") " pod="opendatahub/test-curl-7777-d1d3e3"
Sep 17 14:26:44.578506 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:44.578480    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7777-d1d3e3"
Sep 17 14:26:44.854487 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:44.850023    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7777-d1d3e3"]
Sep 17 14:26:44.858095 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:26:44.858064    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8538c3b5_c914_49af_a938_e694ae1a4842.slice/crio-e1563875d083dc950d1dd2457a26c1acf589b9d1440fa22388c78a316ec39c5e WatchSource:0}: Error finding container e1563875d083dc950d1dd2457a26c1acf589b9d1440fa22388c78a316ec39c5e: Status 404 returned error can't find the container with id e1563875d083dc950d1dd2457a26c1acf589b9d1440fa22388c78a316ec39c5e
Sep 17 14:26:45.836033 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:45.835995    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7777-d1d3e3" event={"ID":"8538c3b5-c914-49af-a938-e694ae1a4842","Type":"ContainerStarted","Data":"7d63d1dc9ed23bf4e7346df3573cdd3448ee2fb2ea241c35ecc013223622d5bd"}
Sep 17 14:26:45.836402 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:45.836041    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7777-d1d3e3" event={"ID":"8538c3b5-c914-49af-a938-e694ae1a4842","Type":"ContainerStarted","Data":"e1563875d083dc950d1dd2457a26c1acf589b9d1440fa22388c78a316ec39c5e"}
Sep 17 14:26:45.851323 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:45.851281    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7777-d1d3e3" podStartSLOduration=1.8512658100000001 podStartE2EDuration="1.85126581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:26:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:26:45.850528844 +0000 UTC m=+1874.792672489" watchObservedRunningTime="2026-09-17 14:26:45.85126581 +0000 UTC m=+1874.793409455"
Sep 17 14:26:46.458059 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:46.458027    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7777-d1d3e3"]
Sep 17 14:26:46.463879 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:46.463851    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7777-d1d3e3"]
Sep 17 14:26:46.842122 ip-10-0-129-158 kubenswrapper[2560]: E0917 14:26:46.842094    2560 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7777-d1d3e3" container="test-curl-7777-d1d3e3"
Sep 17 14:26:47.821190 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:47.821161    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:26:47.847878 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:47.847849    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7777-d1d3e3" podUID="8538c3b5-c914-49af-a938-e694ae1a4842" containerName="test-curl-7777-d1d3e3" containerID="cri-o://7d63d1dc9ed23bf4e7346df3573cdd3448ee2fb2ea241c35ecc013223622d5bd" gracePeriod=2
Sep 17 14:26:49.569356 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:49.569321    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b79988/auth-test-model-b79988-kserve-68d9ff4cbc-xlzsm"]
Sep 17 14:26:49.584115 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:49.584078    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:26:50.021043 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:50.021019    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7777-d1d3e3"
Sep 17 14:26:50.115009 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:50.114954    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pqh2w\" (UniqueName: \"kubernetes.io/projected/8538c3b5-c914-49af-a938-e694ae1a4842-kube-api-access-pqh2w\") pod \"8538c3b5-c914-49af-a938-e694ae1a4842\" (UID: \"8538c3b5-c914-49af-a938-e694ae1a4842\") "
Sep 17 14:26:50.116858 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:50.116835    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8538c3b5-c914-49af-a938-e694ae1a4842-kube-api-access-pqh2w" (OuterVolumeSpecName: "kube-api-access-pqh2w") pod "8538c3b5-c914-49af-a938-e694ae1a4842" (UID: "8538c3b5-c914-49af-a938-e694ae1a4842"). InnerVolumeSpecName "kube-api-access-pqh2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:26:50.216544 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:50.216524    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqh2w\" (UniqueName: \"kubernetes.io/projected/8538c3b5-c914-49af-a938-e694ae1a4842-kube-api-access-pqh2w\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:26:50.868322 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:50.868292    2560 generic.go:356] "Generic (PLEG): container finished" podID="8538c3b5-c914-49af-a938-e694ae1a4842" containerID="7d63d1dc9ed23bf4e7346df3573cdd3448ee2fb2ea241c35ecc013223622d5bd" exitCode=137
Sep 17 14:26:50.868757 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:50.868332    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7777-d1d3e3"
Sep 17 14:26:50.868757 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:50.868400    2560 scope.go:117] "RemoveContainer" containerID="7d63d1dc9ed23bf4e7346df3573cdd3448ee2fb2ea241c35ecc013223622d5bd"
Sep 17 14:26:50.879044 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:50.879028    2560 scope.go:117] "RemoveContainer" containerID="7d63d1dc9ed23bf4e7346df3573cdd3448ee2fb2ea241c35ecc013223622d5bd"
Sep 17 14:26:50.879294 ip-10-0-129-158 kubenswrapper[2560]: E0917 14:26:50.879274    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d63d1dc9ed23bf4e7346df3573cdd3448ee2fb2ea241c35ecc013223622d5bd\": container with ID starting with 7d63d1dc9ed23bf4e7346df3573cdd3448ee2fb2ea241c35ecc013223622d5bd not found: ID does not exist" containerID="7d63d1dc9ed23bf4e7346df3573cdd3448ee2fb2ea241c35ecc013223622d5bd"
Sep 17 14:26:50.879361 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:50.879305    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d63d1dc9ed23bf4e7346df3573cdd3448ee2fb2ea241c35ecc013223622d5bd"} err="failed to get container status \"7d63d1dc9ed23bf4e7346df3573cdd3448ee2fb2ea241c35ecc013223622d5bd\": rpc error: code = NotFound desc = could not find container \"7d63d1dc9ed23bf4e7346df3573cdd3448ee2fb2ea241c35ecc013223622d5bd\": container with ID starting with 7d63d1dc9ed23bf4e7346df3573cdd3448ee2fb2ea241c35ecc013223622d5bd not found: ID does not exist"
Sep 17 14:26:50.880685 ip-10-0-129-158 kubenswrapper[2560]: E0917 14:26:50.880662    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7777-d1d3e3\" is forbidden: User \"system:node:ip-10-0-129-158.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-158.ec2.internal' and this object" podUID="8538c3b5-c914-49af-a938-e694ae1a4842" pod="opendatahub/test-curl-7777-d1d3e3"
Sep 17 14:26:51.568420 ip-10-0-129-158 kubenswrapper[2560]: E0917 14:26:51.568378    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7777-d1d3e3\" is forbidden: User \"system:node:ip-10-0-129-158.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-158.ec2.internal' and this object" podUID="8538c3b5-c914-49af-a938-e694ae1a4842" pod="opendatahub/test-curl-7777-d1d3e3"
Sep 17 14:26:51.569647 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:51.569627    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8538c3b5-c914-49af-a938-e694ae1a4842" path="/var/lib/kubelet/pods/8538c3b5-c914-49af-a938-e694ae1a4842/volumes"
Sep 17 14:26:51.744229 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:51.744196    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:26:51.845715 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:51.845640    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:26:53.637055 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:53.637014    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-b88b90/sub-test-model-b88b90-kserve-67766dbb7c-jsqrx"]
Sep 17 14:26:53.637662 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:53.637642    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8538c3b5-c914-49af-a938-e694ae1a4842" containerName="test-curl-7777-d1d3e3"
Sep 17 14:26:53.637752 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:53.637664    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8538c3b5-c914-49af-a938-e694ae1a4842" containerName="test-curl-7777-d1d3e3"
Sep 17 14:26:53.637812 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:53.637758    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="c66fc21d-4519-4293-bc1f-4e9ae8fe2564" containerName="istio-proxy"
Sep 17 14:26:53.637812 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:53.637777    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="8538c3b5-c914-49af-a938-e694ae1a4842" containerName="test-curl-7777-d1d3e3"
Sep 17 14:26:53.642311 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:53.642291    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-b88b90/sub-test-model-b88b90-kserve-67766dbb7c-jsqrx"
Sep 17 14:26:53.644883 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:53.644865    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-b88b90\"/\"kube-root-ca.crt\""
Sep 17 14:26:53.645594 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:53.645576    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-b88b90\"/\"default-dockercfg-l84qx\""
Sep 17 14:26:53.645645 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:53.645610    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-b88b90\"/\"openshift-service-ca.crt\""
Sep 17 14:26:53.645645 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:53.645621    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-b88b90\"/\"sub-test-model-b88b90-kserve-self-signed-certs\""
Sep 17 14:26:53.652037 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:53.652013    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b88b90/sub-test-model-b88b90-kserve-67766dbb7c-jsqrx"]
Sep 17 14:26:53.705539 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:53.705511    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:26:53.746790 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:53.746762    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a057ca78-151c-430c-b15e-15cd4567d9ae-dshm\") pod \"sub-test-model-b88b90-kserve-67766dbb7c-jsqrx\" (UID: \"a057ca78-151c-430c-b15e-15cd4567d9ae\") " pod="ai-tenant-e2e-shared-a-w3-b88b90/sub-test-model-b88b90-kserve-67766dbb7c-jsqrx"
Sep 17 14:26:53.746904 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:53.746826    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a057ca78-151c-430c-b15e-15cd4567d9ae-tmp-dir\") pod \"sub-test-model-b88b90-kserve-67766dbb7c-jsqrx\" (UID: \"a057ca78-151c-430c-b15e-15cd4567d9ae\") " pod="ai-tenant-e2e-shared-a-w3-b88b90/sub-test-model-b88b90-kserve-67766dbb7c-jsqrx"
Sep 17 14:26:53.746904 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:53.746844    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58bwb\" (UniqueName: \"kubernetes.io/projected/a057ca78-151c-430c-b15e-15cd4567d9ae-kube-api-access-58bwb\") pod \"sub-test-model-b88b90-kserve-67766dbb7c-jsqrx\" (UID: \"a057ca78-151c-430c-b15e-15cd4567d9ae\") " pod="ai-tenant-e2e-shared-a-w3-b88b90/sub-test-model-b88b90-kserve-67766dbb7c-jsqrx"
Sep 17 14:26:53.746904 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:53.746864    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a057ca78-151c-430c-b15e-15cd4567d9ae-tls-certs\") pod \"sub-test-model-b88b90-kserve-67766dbb7c-jsqrx\" (UID: \"a057ca78-151c-430c-b15e-15cd4567d9ae\") " pod="ai-tenant-e2e-shared-a-w3-b88b90/sub-test-model-b88b90-kserve-67766dbb7c-jsqrx"
Sep 17 14:26:53.747008 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:53.746967    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a057ca78-151c-430c-b15e-15cd4567d9ae-home\") pod \"sub-test-model-b88b90-kserve-67766dbb7c-jsqrx\" (UID: \"a057ca78-151c-430c-b15e-15cd4567d9ae\") " pod="ai-tenant-e2e-shared-a-w3-b88b90/sub-test-model-b88b90-kserve-67766dbb7c-jsqrx"
Sep 17 14:26:53.747044 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:53.747016    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a057ca78-151c-430c-b15e-15cd4567d9ae-model-cache\") pod \"sub-test-model-b88b90-kserve-67766dbb7c-jsqrx\" (UID: \"a057ca78-151c-430c-b15e-15cd4567d9ae\") " pod="ai-tenant-e2e-shared-a-w3-b88b90/sub-test-model-b88b90-kserve-67766dbb7c-jsqrx"
Sep 17 14:26:53.847762 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:53.847732    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a057ca78-151c-430c-b15e-15cd4567d9ae-home\") pod \"sub-test-model-b88b90-kserve-67766dbb7c-jsqrx\" (UID: \"a057ca78-151c-430c-b15e-15cd4567d9ae\") " pod="ai-tenant-e2e-shared-a-w3-b88b90/sub-test-model-b88b90-kserve-67766dbb7c-jsqrx"
Sep 17 14:26:53.847891 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:53.847782    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a057ca78-151c-430c-b15e-15cd4567d9ae-model-cache\") pod \"sub-test-model-b88b90-kserve-67766dbb7c-jsqrx\" (UID: \"a057ca78-151c-430c-b15e-15cd4567d9ae\") " pod="ai-tenant-e2e-shared-a-w3-b88b90/sub-test-model-b88b90-kserve-67766dbb7c-jsqrx"
Sep 17 14:26:53.847891 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:53.847808    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a057ca78-151c-430c-b15e-15cd4567d9ae-dshm\") pod \"sub-test-model-b88b90-kserve-67766dbb7c-jsqrx\" (UID: \"a057ca78-151c-430c-b15e-15cd4567d9ae\") " pod="ai-tenant-e2e-shared-a-w3-b88b90/sub-test-model-b88b90-kserve-67766dbb7c-jsqrx"
Sep 17 14:26:53.847891 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:53.847847    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a057ca78-151c-430c-b15e-15cd4567d9ae-tmp-dir\") pod \"sub-test-model-b88b90-kserve-67766dbb7c-jsqrx\" (UID: \"a057ca78-151c-430c-b15e-15cd4567d9ae\") " pod="ai-tenant-e2e-shared-a-w3-b88b90/sub-test-model-b88b90-kserve-67766dbb7c-jsqrx"
Sep 17 14:26:53.847891 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:53.847866    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58bwb\" (UniqueName: \"kubernetes.io/projected/a057ca78-151c-430c-b15e-15cd4567d9ae-kube-api-access-58bwb\") pod \"sub-test-model-b88b90-kserve-67766dbb7c-jsqrx\" (UID: \"a057ca78-151c-430c-b15e-15cd4567d9ae\") " pod="ai-tenant-e2e-shared-a-w3-b88b90/sub-test-model-b88b90-kserve-67766dbb7c-jsqrx"
Sep 17 14:26:53.847891 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:53.847884    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a057ca78-151c-430c-b15e-15cd4567d9ae-tls-certs\") pod \"sub-test-model-b88b90-kserve-67766dbb7c-jsqrx\" (UID: \"a057ca78-151c-430c-b15e-15cd4567d9ae\") " pod="ai-tenant-e2e-shared-a-w3-b88b90/sub-test-model-b88b90-kserve-67766dbb7c-jsqrx"
Sep 17 14:26:53.848207 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:53.848173    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a057ca78-151c-430c-b15e-15cd4567d9ae-home\") pod \"sub-test-model-b88b90-kserve-67766dbb7c-jsqrx\" (UID: \"a057ca78-151c-430c-b15e-15cd4567d9ae\") " pod="ai-tenant-e2e-shared-a-w3-b88b90/sub-test-model-b88b90-kserve-67766dbb7c-jsqrx"
Sep 17 14:26:53.848317 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:53.848210    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a057ca78-151c-430c-b15e-15cd4567d9ae-model-cache\") pod \"sub-test-model-b88b90-kserve-67766dbb7c-jsqrx\" (UID: \"a057ca78-151c-430c-b15e-15cd4567d9ae\") " pod="ai-tenant-e2e-shared-a-w3-b88b90/sub-test-model-b88b90-kserve-67766dbb7c-jsqrx"
Sep 17 14:26:53.848360 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:53.848330    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a057ca78-151c-430c-b15e-15cd4567d9ae-tmp-dir\") pod \"sub-test-model-b88b90-kserve-67766dbb7c-jsqrx\" (UID: \"a057ca78-151c-430c-b15e-15cd4567d9ae\") " pod="ai-tenant-e2e-shared-a-w3-b88b90/sub-test-model-b88b90-kserve-67766dbb7c-jsqrx"
Sep 17 14:26:53.850009 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:53.849989    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a057ca78-151c-430c-b15e-15cd4567d9ae-dshm\") pod \"sub-test-model-b88b90-kserve-67766dbb7c-jsqrx\" (UID: \"a057ca78-151c-430c-b15e-15cd4567d9ae\") " pod="ai-tenant-e2e-shared-a-w3-b88b90/sub-test-model-b88b90-kserve-67766dbb7c-jsqrx"
Sep 17 14:26:53.850317 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:53.850298    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a057ca78-151c-430c-b15e-15cd4567d9ae-tls-certs\") pod \"sub-test-model-b88b90-kserve-67766dbb7c-jsqrx\" (UID: \"a057ca78-151c-430c-b15e-15cd4567d9ae\") " pod="ai-tenant-e2e-shared-a-w3-b88b90/sub-test-model-b88b90-kserve-67766dbb7c-jsqrx"
Sep 17 14:26:53.856050 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:53.856030    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58bwb\" (UniqueName: \"kubernetes.io/projected/a057ca78-151c-430c-b15e-15cd4567d9ae-kube-api-access-58bwb\") pod \"sub-test-model-b88b90-kserve-67766dbb7c-jsqrx\" (UID: \"a057ca78-151c-430c-b15e-15cd4567d9ae\") " pod="ai-tenant-e2e-shared-a-w3-b88b90/sub-test-model-b88b90-kserve-67766dbb7c-jsqrx"
Sep 17 14:26:53.952794 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:53.952716    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-b88b90/sub-test-model-b88b90-kserve-67766dbb7c-jsqrx"
Sep 17 14:26:54.228411 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:54.228383    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b88b90/sub-test-model-b88b90-kserve-67766dbb7c-jsqrx"]
Sep 17 14:26:54.230554 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:26:54.230529    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda057ca78_151c_430c_b15e_15cd4567d9ae.slice/crio-4294df43f9c8eab731d39e6737c7786954a23f430ed7d5d425c62688262d1e9c WatchSource:0}: Error finding container 4294df43f9c8eab731d39e6737c7786954a23f430ed7d5d425c62688262d1e9c: Status 404 returned error can't find the container with id 4294df43f9c8eab731d39e6737c7786954a23f430ed7d5d425c62688262d1e9c
Sep 17 14:26:54.897887 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:54.897849    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-b88b90/sub-test-model-b88b90-kserve-67766dbb7c-jsqrx" event={"ID":"a057ca78-151c-430c-b15e-15cd4567d9ae","Type":"ContainerStarted","Data":"3ffad95d55c0cc638abf48420c7b6ecd802a4e7429c0ad102663607dcbfabcfe"}
Sep 17 14:26:54.898240 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:54.897896    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-b88b90/sub-test-model-b88b90-kserve-67766dbb7c-jsqrx" event={"ID":"a057ca78-151c-430c-b15e-15cd4567d9ae","Type":"ContainerStarted","Data":"4294df43f9c8eab731d39e6737c7786954a23f430ed7d5d425c62688262d1e9c"}
Sep 17 14:26:54.918231 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:54.918186    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-b88b90/sub-test-model-b88b90-kserve-67766dbb7c-jsqrx" podStartSLOduration=1.918172457 podStartE2EDuration="1.918172457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:26:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:26:54.915492299 +0000 UTC m=+1883.857635936" watchObservedRunningTime="2026-09-17 14:26:54.918172457 +0000 UTC m=+1883.860316104"
Sep 17 14:26:55.670257 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:55.670222    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:26:55.767400 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:55.767366    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:26:57.246600 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:26:57.246565    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:03.953629 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:03.953594    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-b88b90/sub-test-model-b88b90-kserve-67766dbb7c-jsqrx"
Sep 17 14:27:03.953629 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:03.953639    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-b88b90/sub-test-model-b88b90-kserve-67766dbb7c-jsqrx"
Sep 17 14:27:03.965904 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:03.965881    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-b88b90/sub-test-model-b88b90-kserve-67766dbb7c-jsqrx"
Sep 17 14:27:04.026645 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:04.026588    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:04.588494 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:04.588437    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-b79988/auth-test-model-b79988-kserve-68d9ff4cbc-xlzsm" podUID="9b239de1-635b-4718-8349-6e5d5bf9f506" containerName="main" containerID="cri-o://1518bbb163fbe167a109a4328ed1bd0f8a4fa33d236d59ef74c068890890b7b1" gracePeriod=45
Sep 17 14:27:04.841948 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:04.841895    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b79988/auth-test-model-b79988-kserve-68d9ff4cbc-xlzsm"
Sep 17 14:27:04.946604 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:04.946572    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b239de1-635b-4718-8349-6e5d5bf9f506-dshm\") pod \"9b239de1-635b-4718-8349-6e5d5bf9f506\" (UID: \"9b239de1-635b-4718-8349-6e5d5bf9f506\") "
Sep 17 14:27:04.946767 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:04.946613    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b239de1-635b-4718-8349-6e5d5bf9f506-tmp-dir\") pod \"9b239de1-635b-4718-8349-6e5d5bf9f506\" (UID: \"9b239de1-635b-4718-8349-6e5d5bf9f506\") "
Sep 17 14:27:04.946767 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:04.946667    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b239de1-635b-4718-8349-6e5d5bf9f506-model-cache\") pod \"9b239de1-635b-4718-8349-6e5d5bf9f506\" (UID: \"9b239de1-635b-4718-8349-6e5d5bf9f506\") "
Sep 17 14:27:04.946767 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:04.946705    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b239de1-635b-4718-8349-6e5d5bf9f506-tls-certs\") pod \"9b239de1-635b-4718-8349-6e5d5bf9f506\" (UID: \"9b239de1-635b-4718-8349-6e5d5bf9f506\") "
Sep 17 14:27:04.946905 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:04.946802    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b239de1-635b-4718-8349-6e5d5bf9f506-home\") pod \"9b239de1-635b-4718-8349-6e5d5bf9f506\" (UID: \"9b239de1-635b-4718-8349-6e5d5bf9f506\") "
Sep 17 14:27:04.946967 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:04.946906    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5ht4d\" (UniqueName: \"kubernetes.io/projected/9b239de1-635b-4718-8349-6e5d5bf9f506-kube-api-access-5ht4d\") pod \"9b239de1-635b-4718-8349-6e5d5bf9f506\" (UID: \"9b239de1-635b-4718-8349-6e5d5bf9f506\") "
Sep 17 14:27:04.946967 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:04.946951    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b239de1-635b-4718-8349-6e5d5bf9f506-model-cache" (OuterVolumeSpecName: "model-cache") pod "9b239de1-635b-4718-8349-6e5d5bf9f506" (UID: "9b239de1-635b-4718-8349-6e5d5bf9f506"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:27:04.947081 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:04.946962    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b239de1-635b-4718-8349-6e5d5bf9f506-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9b239de1-635b-4718-8349-6e5d5bf9f506" (UID: "9b239de1-635b-4718-8349-6e5d5bf9f506"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:27:04.947143 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:04.947095    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b239de1-635b-4718-8349-6e5d5bf9f506-home" (OuterVolumeSpecName: "home") pod "9b239de1-635b-4718-8349-6e5d5bf9f506" (UID: "9b239de1-635b-4718-8349-6e5d5bf9f506"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:27:04.947318 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:04.947292    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b239de1-635b-4718-8349-6e5d5bf9f506-tmp-dir\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:27:04.947318 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:04.947320    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b239de1-635b-4718-8349-6e5d5bf9f506-model-cache\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:27:04.947539 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:04.947332    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b239de1-635b-4718-8349-6e5d5bf9f506-home\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:27:04.948931 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:04.948902    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b239de1-635b-4718-8349-6e5d5bf9f506-kube-api-access-5ht4d" (OuterVolumeSpecName: "kube-api-access-5ht4d") pod "9b239de1-635b-4718-8349-6e5d5bf9f506" (UID: "9b239de1-635b-4718-8349-6e5d5bf9f506"). InnerVolumeSpecName "kube-api-access-5ht4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:27:04.948931 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:04.948925    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b239de1-635b-4718-8349-6e5d5bf9f506-dshm" (OuterVolumeSpecName: "dshm") pod "9b239de1-635b-4718-8349-6e5d5bf9f506" (UID: "9b239de1-635b-4718-8349-6e5d5bf9f506"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:27:04.949052 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:04.948976    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b239de1-635b-4718-8349-6e5d5bf9f506-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9b239de1-635b-4718-8349-6e5d5bf9f506" (UID: "9b239de1-635b-4718-8349-6e5d5bf9f506"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:27:04.967223 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:04.967199    2560 generic.go:356] "Generic (PLEG): container finished" podID="9b239de1-635b-4718-8349-6e5d5bf9f506" containerID="1518bbb163fbe167a109a4328ed1bd0f8a4fa33d236d59ef74c068890890b7b1" exitCode=0
Sep 17 14:27:04.967525 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:04.967243    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b79988/auth-test-model-b79988-kserve-68d9ff4cbc-xlzsm"
Sep 17 14:27:04.967525 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:04.967273    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b79988/auth-test-model-b79988-kserve-68d9ff4cbc-xlzsm" event={"ID":"9b239de1-635b-4718-8349-6e5d5bf9f506","Type":"ContainerDied","Data":"1518bbb163fbe167a109a4328ed1bd0f8a4fa33d236d59ef74c068890890b7b1"}
Sep 17 14:27:04.967525 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:04.967312    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b79988/auth-test-model-b79988-kserve-68d9ff4cbc-xlzsm" event={"ID":"9b239de1-635b-4718-8349-6e5d5bf9f506","Type":"ContainerDied","Data":"7b5051dff7ba6304a317c9eeaa168b728f7482e35ccd562ae3b0218eda701404"}
Sep 17 14:27:04.967525 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:04.967331    2560 scope.go:117] "RemoveContainer" containerID="1518bbb163fbe167a109a4328ed1bd0f8a4fa33d236d59ef74c068890890b7b1"
Sep 17 14:27:04.979726 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:04.979710    2560 scope.go:117] "RemoveContainer" containerID="1518bbb163fbe167a109a4328ed1bd0f8a4fa33d236d59ef74c068890890b7b1"
Sep 17 14:27:04.980023 ip-10-0-129-158 kubenswrapper[2560]: E0917 14:27:04.980003    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1518bbb163fbe167a109a4328ed1bd0f8a4fa33d236d59ef74c068890890b7b1\": container with ID starting with 1518bbb163fbe167a109a4328ed1bd0f8a4fa33d236d59ef74c068890890b7b1 not found: ID does not exist" containerID="1518bbb163fbe167a109a4328ed1bd0f8a4fa33d236d59ef74c068890890b7b1"
Sep 17 14:27:04.980086 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:04.980032    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1518bbb163fbe167a109a4328ed1bd0f8a4fa33d236d59ef74c068890890b7b1"} err="failed to get container status \"1518bbb163fbe167a109a4328ed1bd0f8a4fa33d236d59ef74c068890890b7b1\": rpc error: code = NotFound desc = could not find container \"1518bbb163fbe167a109a4328ed1bd0f8a4fa33d236d59ef74c068890890b7b1\": container with ID starting with 1518bbb163fbe167a109a4328ed1bd0f8a4fa33d236d59ef74c068890890b7b1 not found: ID does not exist"
Sep 17 14:27:04.981179 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:04.981164    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-b88b90/sub-test-model-b88b90-kserve-67766dbb7c-jsqrx"
Sep 17 14:27:04.992384 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:04.992364    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b79988/auth-test-model-b79988-kserve-68d9ff4cbc-xlzsm"]
Sep 17 14:27:04.998259 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:04.998241    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b79988/auth-test-model-b79988-kserve-68d9ff4cbc-xlzsm"]
Sep 17 14:27:05.048026 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:05.048002    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b239de1-635b-4718-8349-6e5d5bf9f506-tls-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:27:05.048026 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:05.048027    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ht4d\" (UniqueName: \"kubernetes.io/projected/9b239de1-635b-4718-8349-6e5d5bf9f506-kube-api-access-5ht4d\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:27:05.048188 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:05.048039    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b239de1-635b-4718-8349-6e5d5bf9f506-dshm\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:27:05.569191 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:05.569158    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b239de1-635b-4718-8349-6e5d5bf9f506" path="/var/lib/kubelet/pods/9b239de1-635b-4718-8349-6e5d5bf9f506/volumes"
Sep 17 14:27:19.726692 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:19.726656    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-b79988/sub-test-model-b79988-kserve-8546cd8769-bhdjb"]
Sep 17 14:27:19.727199 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:19.727180    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b239de1-635b-4718-8349-6e5d5bf9f506" containerName="main"
Sep 17 14:27:19.727199 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:19.727199    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b239de1-635b-4718-8349-6e5d5bf9f506" containerName="main"
Sep 17 14:27:19.727302 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:19.727287    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b239de1-635b-4718-8349-6e5d5bf9f506" containerName="main"
Sep 17 14:27:19.731619 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:19.731602    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b79988/sub-test-model-b79988-kserve-8546cd8769-bhdjb"
Sep 17 14:27:19.734628 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:19.734605    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-b79988\"/\"kube-root-ca.crt\""
Sep 17 14:27:19.734780 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:19.734672    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-b79988\"/\"sub-test-model-b79988-kserve-self-signed-certs\""
Sep 17 14:27:19.734780 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:19.734687    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-b79988\"/\"default-dockercfg-gjn9m\""
Sep 17 14:27:19.734921 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:19.734907    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-b79988\"/\"openshift-service-ca.crt\""
Sep 17 14:27:19.746804 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:19.746778    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b79988/sub-test-model-b79988-kserve-8546cd8769-bhdjb"]
Sep 17 14:27:19.757390 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:19.757364    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:19.872088 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:19.872064    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6ce7cee-ce2a-4bba-938c-92dde9e042bc-dshm\") pod \"sub-test-model-b79988-kserve-8546cd8769-bhdjb\" (UID: \"c6ce7cee-ce2a-4bba-938c-92dde9e042bc\") " pod="ai-tenant-e2e-shared-b-w3-b79988/sub-test-model-b79988-kserve-8546cd8769-bhdjb"
Sep 17 14:27:19.872238 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:19.872115    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6ce7cee-ce2a-4bba-938c-92dde9e042bc-home\") pod \"sub-test-model-b79988-kserve-8546cd8769-bhdjb\" (UID: \"c6ce7cee-ce2a-4bba-938c-92dde9e042bc\") " pod="ai-tenant-e2e-shared-b-w3-b79988/sub-test-model-b79988-kserve-8546cd8769-bhdjb"
Sep 17 14:27:19.872238 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:19.872146    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6ce7cee-ce2a-4bba-938c-92dde9e042bc-tmp-dir\") pod \"sub-test-model-b79988-kserve-8546cd8769-bhdjb\" (UID: \"c6ce7cee-ce2a-4bba-938c-92dde9e042bc\") " pod="ai-tenant-e2e-shared-b-w3-b79988/sub-test-model-b79988-kserve-8546cd8769-bhdjb"
Sep 17 14:27:19.872238 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:19.872174    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6ce7cee-ce2a-4bba-938c-92dde9e042bc-tls-certs\") pod \"sub-test-model-b79988-kserve-8546cd8769-bhdjb\" (UID: \"c6ce7cee-ce2a-4bba-938c-92dde9e042bc\") " pod="ai-tenant-e2e-shared-b-w3-b79988/sub-test-model-b79988-kserve-8546cd8769-bhdjb"
Sep 17 14:27:19.872238 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:19.872199    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6ce7cee-ce2a-4bba-938c-92dde9e042bc-model-cache\") pod \"sub-test-model-b79988-kserve-8546cd8769-bhdjb\" (UID: \"c6ce7cee-ce2a-4bba-938c-92dde9e042bc\") " pod="ai-tenant-e2e-shared-b-w3-b79988/sub-test-model-b79988-kserve-8546cd8769-bhdjb"
Sep 17 14:27:19.872401 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:19.872320    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw7rv\" (UniqueName: \"kubernetes.io/projected/c6ce7cee-ce2a-4bba-938c-92dde9e042bc-kube-api-access-mw7rv\") pod \"sub-test-model-b79988-kserve-8546cd8769-bhdjb\" (UID: \"c6ce7cee-ce2a-4bba-938c-92dde9e042bc\") " pod="ai-tenant-e2e-shared-b-w3-b79988/sub-test-model-b79988-kserve-8546cd8769-bhdjb"
Sep 17 14:27:19.973144 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:19.973115    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw7rv\" (UniqueName: \"kubernetes.io/projected/c6ce7cee-ce2a-4bba-938c-92dde9e042bc-kube-api-access-mw7rv\") pod \"sub-test-model-b79988-kserve-8546cd8769-bhdjb\" (UID: \"c6ce7cee-ce2a-4bba-938c-92dde9e042bc\") " pod="ai-tenant-e2e-shared-b-w3-b79988/sub-test-model-b79988-kserve-8546cd8769-bhdjb"
Sep 17 14:27:19.973249 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:19.973151    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6ce7cee-ce2a-4bba-938c-92dde9e042bc-dshm\") pod \"sub-test-model-b79988-kserve-8546cd8769-bhdjb\" (UID: \"c6ce7cee-ce2a-4bba-938c-92dde9e042bc\") " pod="ai-tenant-e2e-shared-b-w3-b79988/sub-test-model-b79988-kserve-8546cd8769-bhdjb"
Sep 17 14:27:19.973249 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:19.973179    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6ce7cee-ce2a-4bba-938c-92dde9e042bc-home\") pod \"sub-test-model-b79988-kserve-8546cd8769-bhdjb\" (UID: \"c6ce7cee-ce2a-4bba-938c-92dde9e042bc\") " pod="ai-tenant-e2e-shared-b-w3-b79988/sub-test-model-b79988-kserve-8546cd8769-bhdjb"
Sep 17 14:27:19.973249 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:19.973206    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6ce7cee-ce2a-4bba-938c-92dde9e042bc-tmp-dir\") pod \"sub-test-model-b79988-kserve-8546cd8769-bhdjb\" (UID: \"c6ce7cee-ce2a-4bba-938c-92dde9e042bc\") " pod="ai-tenant-e2e-shared-b-w3-b79988/sub-test-model-b79988-kserve-8546cd8769-bhdjb"
Sep 17 14:27:19.973249 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:19.973224    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6ce7cee-ce2a-4bba-938c-92dde9e042bc-tls-certs\") pod \"sub-test-model-b79988-kserve-8546cd8769-bhdjb\" (UID: \"c6ce7cee-ce2a-4bba-938c-92dde9e042bc\") " pod="ai-tenant-e2e-shared-b-w3-b79988/sub-test-model-b79988-kserve-8546cd8769-bhdjb"
Sep 17 14:27:19.973249 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:19.973247    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6ce7cee-ce2a-4bba-938c-92dde9e042bc-model-cache\") pod \"sub-test-model-b79988-kserve-8546cd8769-bhdjb\" (UID: \"c6ce7cee-ce2a-4bba-938c-92dde9e042bc\") " pod="ai-tenant-e2e-shared-b-w3-b79988/sub-test-model-b79988-kserve-8546cd8769-bhdjb"
Sep 17 14:27:19.973620 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:19.973596    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6ce7cee-ce2a-4bba-938c-92dde9e042bc-home\") pod \"sub-test-model-b79988-kserve-8546cd8769-bhdjb\" (UID: \"c6ce7cee-ce2a-4bba-938c-92dde9e042bc\") " pod="ai-tenant-e2e-shared-b-w3-b79988/sub-test-model-b79988-kserve-8546cd8769-bhdjb"
Sep 17 14:27:19.973620 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:19.973614    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6ce7cee-ce2a-4bba-938c-92dde9e042bc-tmp-dir\") pod \"sub-test-model-b79988-kserve-8546cd8769-bhdjb\" (UID: \"c6ce7cee-ce2a-4bba-938c-92dde9e042bc\") " pod="ai-tenant-e2e-shared-b-w3-b79988/sub-test-model-b79988-kserve-8546cd8769-bhdjb"
Sep 17 14:27:19.973743 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:19.973674    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6ce7cee-ce2a-4bba-938c-92dde9e042bc-model-cache\") pod \"sub-test-model-b79988-kserve-8546cd8769-bhdjb\" (UID: \"c6ce7cee-ce2a-4bba-938c-92dde9e042bc\") " pod="ai-tenant-e2e-shared-b-w3-b79988/sub-test-model-b79988-kserve-8546cd8769-bhdjb"
Sep 17 14:27:19.975337 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:19.975321    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6ce7cee-ce2a-4bba-938c-92dde9e042bc-dshm\") pod \"sub-test-model-b79988-kserve-8546cd8769-bhdjb\" (UID: \"c6ce7cee-ce2a-4bba-938c-92dde9e042bc\") " pod="ai-tenant-e2e-shared-b-w3-b79988/sub-test-model-b79988-kserve-8546cd8769-bhdjb"
Sep 17 14:27:19.975680 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:19.975662    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6ce7cee-ce2a-4bba-938c-92dde9e042bc-tls-certs\") pod \"sub-test-model-b79988-kserve-8546cd8769-bhdjb\" (UID: \"c6ce7cee-ce2a-4bba-938c-92dde9e042bc\") " pod="ai-tenant-e2e-shared-b-w3-b79988/sub-test-model-b79988-kserve-8546cd8769-bhdjb"
Sep 17 14:27:19.984648 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:19.984600    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw7rv\" (UniqueName: \"kubernetes.io/projected/c6ce7cee-ce2a-4bba-938c-92dde9e042bc-kube-api-access-mw7rv\") pod \"sub-test-model-b79988-kserve-8546cd8769-bhdjb\" (UID: \"c6ce7cee-ce2a-4bba-938c-92dde9e042bc\") " pod="ai-tenant-e2e-shared-b-w3-b79988/sub-test-model-b79988-kserve-8546cd8769-bhdjb"
Sep 17 14:27:20.041270 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:20.041235    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b79988/sub-test-model-b79988-kserve-8546cd8769-bhdjb"
Sep 17 14:27:20.324048 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:20.324023    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b79988/sub-test-model-b79988-kserve-8546cd8769-bhdjb"]
Sep 17 14:27:20.325606 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:27:20.325579    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc6ce7cee_ce2a_4bba_938c_92dde9e042bc.slice/crio-47cd64443054cfd307849c82f0185daf6840e90c23ad521687cd04ee93dcd508 WatchSource:0}: Error finding container 47cd64443054cfd307849c82f0185daf6840e90c23ad521687cd04ee93dcd508: Status 404 returned error can't find the container with id 47cd64443054cfd307849c82f0185daf6840e90c23ad521687cd04ee93dcd508
Sep 17 14:27:20.617265 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:20.617179    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:21.065985 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:21.065955    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b79988/sub-test-model-b79988-kserve-8546cd8769-bhdjb" event={"ID":"c6ce7cee-ce2a-4bba-938c-92dde9e042bc","Type":"ContainerStarted","Data":"da965650654a0fc8e497517b82d693008b5fea961d729f2c8dfe326b8983b216"}
Sep 17 14:27:21.066342 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:21.065996    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b79988/sub-test-model-b79988-kserve-8546cd8769-bhdjb" event={"ID":"c6ce7cee-ce2a-4bba-938c-92dde9e042bc","Type":"ContainerStarted","Data":"47cd64443054cfd307849c82f0185daf6840e90c23ad521687cd04ee93dcd508"}
Sep 17 14:27:21.101955 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:21.101910    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-b79988/sub-test-model-b79988-kserve-8546cd8769-bhdjb" podStartSLOduration=2.101896294 podStartE2EDuration="2.101896294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:27:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:27:21.101267037 +0000 UTC m=+1910.043410683" watchObservedRunningTime="2026-09-17 14:27:21.101896294 +0000 UTC m=+1910.044039939"
Sep 17 14:27:30.042168 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:30.042137    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-b79988/sub-test-model-b79988-kserve-8546cd8769-bhdjb"
Sep 17 14:27:30.042168 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:30.042178    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-b79988/sub-test-model-b79988-kserve-8546cd8769-bhdjb"
Sep 17 14:27:30.054668 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:30.054646    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-b79988/sub-test-model-b79988-kserve-8546cd8769-bhdjb"
Sep 17 14:27:30.134919 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:30.134895    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-b79988/sub-test-model-b79988-kserve-8546cd8769-bhdjb"
Sep 17 14:27:31.665414 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:31.665380    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:32.074176 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:32.074151    2560 scope.go:117] "RemoveContainer" containerID="b238172f01a8ee478f30d15e87d63c05ccf532d2effe7cad7422e1f4f3e2c892"
Sep 17 14:27:55.499349 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:55.499310    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:56.248090 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:56.248056    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b88b90/sub-test-model-b88b90-kserve-67766dbb7c-jsqrx"]
Sep 17 14:27:56.840450 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:56.840418    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:57.223067 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:57.222995    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b79988/sub-test-model-b79988-kserve-8546cd8769-bhdjb"]
Sep 17 14:27:57.237218 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:27:57.237189    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:28:11.266795 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:11.266734    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-b88b90/sub-test-model-b88b90-kserve-67766dbb7c-jsqrx" podUID="a057ca78-151c-430c-b15e-15cd4567d9ae" containerName="main" containerID="cri-o://3ffad95d55c0cc638abf48420c7b6ecd802a4e7429c0ad102663607dcbfabcfe" gracePeriod=45
Sep 17 14:28:11.372166 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:11.372124    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:28:11.508052 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:11.508029    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-b88b90/sub-test-model-b88b90-kserve-67766dbb7c-jsqrx"
Sep 17 14:28:11.536141 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:11.536073    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a057ca78-151c-430c-b15e-15cd4567d9ae-tls-certs\") pod \"a057ca78-151c-430c-b15e-15cd4567d9ae\" (UID: \"a057ca78-151c-430c-b15e-15cd4567d9ae\") "
Sep 17 14:28:11.536141 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:11.536111    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-58bwb\" (UniqueName: \"kubernetes.io/projected/a057ca78-151c-430c-b15e-15cd4567d9ae-kube-api-access-58bwb\") pod \"a057ca78-151c-430c-b15e-15cd4567d9ae\" (UID: \"a057ca78-151c-430c-b15e-15cd4567d9ae\") "
Sep 17 14:28:11.536322 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:11.536151    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a057ca78-151c-430c-b15e-15cd4567d9ae-home\") pod \"a057ca78-151c-430c-b15e-15cd4567d9ae\" (UID: \"a057ca78-151c-430c-b15e-15cd4567d9ae\") "
Sep 17 14:28:11.536322 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:11.536236    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a057ca78-151c-430c-b15e-15cd4567d9ae-model-cache\") pod \"a057ca78-151c-430c-b15e-15cd4567d9ae\" (UID: \"a057ca78-151c-430c-b15e-15cd4567d9ae\") "
Sep 17 14:28:11.536322 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:11.536314    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a057ca78-151c-430c-b15e-15cd4567d9ae-tmp-dir\") pod \"a057ca78-151c-430c-b15e-15cd4567d9ae\" (UID: \"a057ca78-151c-430c-b15e-15cd4567d9ae\") "
Sep 17 14:28:11.536533 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:11.536348    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a057ca78-151c-430c-b15e-15cd4567d9ae-home" (OuterVolumeSpecName: "home") pod "a057ca78-151c-430c-b15e-15cd4567d9ae" (UID: "a057ca78-151c-430c-b15e-15cd4567d9ae"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:28:11.536533 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:11.536358    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a057ca78-151c-430c-b15e-15cd4567d9ae-dshm\") pod \"a057ca78-151c-430c-b15e-15cd4567d9ae\" (UID: \"a057ca78-151c-430c-b15e-15cd4567d9ae\") "
Sep 17 14:28:11.536533 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:11.536479    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a057ca78-151c-430c-b15e-15cd4567d9ae-model-cache" (OuterVolumeSpecName: "model-cache") pod "a057ca78-151c-430c-b15e-15cd4567d9ae" (UID: "a057ca78-151c-430c-b15e-15cd4567d9ae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:28:11.536533 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:11.536517    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a057ca78-151c-430c-b15e-15cd4567d9ae-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a057ca78-151c-430c-b15e-15cd4567d9ae" (UID: "a057ca78-151c-430c-b15e-15cd4567d9ae"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:28:11.536758 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:11.536732    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a057ca78-151c-430c-b15e-15cd4567d9ae-home\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:28:11.536758 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:11.536745    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a057ca78-151c-430c-b15e-15cd4567d9ae-model-cache\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:28:11.536758 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:11.536754    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a057ca78-151c-430c-b15e-15cd4567d9ae-tmp-dir\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:28:11.538319 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:11.538299    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a057ca78-151c-430c-b15e-15cd4567d9ae-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a057ca78-151c-430c-b15e-15cd4567d9ae" (UID: "a057ca78-151c-430c-b15e-15cd4567d9ae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:28:11.538426 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:11.538345    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a057ca78-151c-430c-b15e-15cd4567d9ae-kube-api-access-58bwb" (OuterVolumeSpecName: "kube-api-access-58bwb") pod "a057ca78-151c-430c-b15e-15cd4567d9ae" (UID: "a057ca78-151c-430c-b15e-15cd4567d9ae"). InnerVolumeSpecName "kube-api-access-58bwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:28:11.538426 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:11.538412    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a057ca78-151c-430c-b15e-15cd4567d9ae-dshm" (OuterVolumeSpecName: "dshm") pod "a057ca78-151c-430c-b15e-15cd4567d9ae" (UID: "a057ca78-151c-430c-b15e-15cd4567d9ae"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:28:11.638048 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:11.638022    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a057ca78-151c-430c-b15e-15cd4567d9ae-tls-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:28:11.638048 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:11.638046    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58bwb\" (UniqueName: \"kubernetes.io/projected/a057ca78-151c-430c-b15e-15cd4567d9ae-kube-api-access-58bwb\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:28:11.638196 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:11.638055    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a057ca78-151c-430c-b15e-15cd4567d9ae-dshm\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:28:12.242306 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:12.242201    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-b79988/sub-test-model-b79988-kserve-8546cd8769-bhdjb" podUID="c6ce7cee-ce2a-4bba-938c-92dde9e042bc" containerName="main" containerID="cri-o://da965650654a0fc8e497517b82d693008b5fea961d729f2c8dfe326b8983b216" gracePeriod=45
Sep 17 14:28:12.379189 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:12.379159    2560 generic.go:356] "Generic (PLEG): container finished" podID="a057ca78-151c-430c-b15e-15cd4567d9ae" containerID="3ffad95d55c0cc638abf48420c7b6ecd802a4e7429c0ad102663607dcbfabcfe" exitCode=0
Sep 17 14:28:12.379572 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:12.379214    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-b88b90/sub-test-model-b88b90-kserve-67766dbb7c-jsqrx"
Sep 17 14:28:12.379572 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:12.379242    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-b88b90/sub-test-model-b88b90-kserve-67766dbb7c-jsqrx" event={"ID":"a057ca78-151c-430c-b15e-15cd4567d9ae","Type":"ContainerDied","Data":"3ffad95d55c0cc638abf48420c7b6ecd802a4e7429c0ad102663607dcbfabcfe"}
Sep 17 14:28:12.379572 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:12.379289    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-b88b90/sub-test-model-b88b90-kserve-67766dbb7c-jsqrx" event={"ID":"a057ca78-151c-430c-b15e-15cd4567d9ae","Type":"ContainerDied","Data":"4294df43f9c8eab731d39e6737c7786954a23f430ed7d5d425c62688262d1e9c"}
Sep 17 14:28:12.379572 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:12.379311    2560 scope.go:117] "RemoveContainer" containerID="3ffad95d55c0cc638abf48420c7b6ecd802a4e7429c0ad102663607dcbfabcfe"
Sep 17 14:28:12.381364 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:12.381341    2560 generic.go:356] "Generic (PLEG): container finished" podID="c6ce7cee-ce2a-4bba-938c-92dde9e042bc" containerID="da965650654a0fc8e497517b82d693008b5fea961d729f2c8dfe326b8983b216" exitCode=0
Sep 17 14:28:12.381509 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:12.381381    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b79988/sub-test-model-b79988-kserve-8546cd8769-bhdjb" event={"ID":"c6ce7cee-ce2a-4bba-938c-92dde9e042bc","Type":"ContainerDied","Data":"da965650654a0fc8e497517b82d693008b5fea961d729f2c8dfe326b8983b216"}
Sep 17 14:28:12.389360 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:12.389333    2560 scope.go:117] "RemoveContainer" containerID="3ffad95d55c0cc638abf48420c7b6ecd802a4e7429c0ad102663607dcbfabcfe"
Sep 17 14:28:12.389762 ip-10-0-129-158 kubenswrapper[2560]: E0917 14:28:12.389738    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ffad95d55c0cc638abf48420c7b6ecd802a4e7429c0ad102663607dcbfabcfe\": container with ID starting with 3ffad95d55c0cc638abf48420c7b6ecd802a4e7429c0ad102663607dcbfabcfe not found: ID does not exist" containerID="3ffad95d55c0cc638abf48420c7b6ecd802a4e7429c0ad102663607dcbfabcfe"
Sep 17 14:28:12.389848 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:12.389770    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ffad95d55c0cc638abf48420c7b6ecd802a4e7429c0ad102663607dcbfabcfe"} err="failed to get container status \"3ffad95d55c0cc638abf48420c7b6ecd802a4e7429c0ad102663607dcbfabcfe\": rpc error: code = NotFound desc = could not find container \"3ffad95d55c0cc638abf48420c7b6ecd802a4e7429c0ad102663607dcbfabcfe\": container with ID starting with 3ffad95d55c0cc638abf48420c7b6ecd802a4e7429c0ad102663607dcbfabcfe not found: ID does not exist"
Sep 17 14:28:12.413942 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:12.413918    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b88b90/sub-test-model-b88b90-kserve-67766dbb7c-jsqrx"]
Sep 17 14:28:12.429557 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:12.429523    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b88b90/sub-test-model-b88b90-kserve-67766dbb7c-jsqrx"]
Sep 17 14:28:12.491162 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:12.491141    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b79988/sub-test-model-b79988-kserve-8546cd8769-bhdjb"
Sep 17 14:28:12.547022 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:12.546969    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6ce7cee-ce2a-4bba-938c-92dde9e042bc-home\") pod \"c6ce7cee-ce2a-4bba-938c-92dde9e042bc\" (UID: \"c6ce7cee-ce2a-4bba-938c-92dde9e042bc\") "
Sep 17 14:28:12.547139 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:12.547037    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mw7rv\" (UniqueName: \"kubernetes.io/projected/c6ce7cee-ce2a-4bba-938c-92dde9e042bc-kube-api-access-mw7rv\") pod \"c6ce7cee-ce2a-4bba-938c-92dde9e042bc\" (UID: \"c6ce7cee-ce2a-4bba-938c-92dde9e042bc\") "
Sep 17 14:28:12.547139 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:12.547074    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6ce7cee-ce2a-4bba-938c-92dde9e042bc-model-cache\") pod \"c6ce7cee-ce2a-4bba-938c-92dde9e042bc\" (UID: \"c6ce7cee-ce2a-4bba-938c-92dde9e042bc\") "
Sep 17 14:28:12.547139 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:12.547099    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6ce7cee-ce2a-4bba-938c-92dde9e042bc-tmp-dir\") pod \"c6ce7cee-ce2a-4bba-938c-92dde9e042bc\" (UID: \"c6ce7cee-ce2a-4bba-938c-92dde9e042bc\") "
Sep 17 14:28:12.547317 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:12.547146    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6ce7cee-ce2a-4bba-938c-92dde9e042bc-dshm\") pod \"c6ce7cee-ce2a-4bba-938c-92dde9e042bc\" (UID: \"c6ce7cee-ce2a-4bba-938c-92dde9e042bc\") "
Sep 17 14:28:12.547317 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:12.547175    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6ce7cee-ce2a-4bba-938c-92dde9e042bc-tls-certs\") pod \"c6ce7cee-ce2a-4bba-938c-92dde9e042bc\" (UID: \"c6ce7cee-ce2a-4bba-938c-92dde9e042bc\") "
Sep 17 14:28:12.547317 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:12.547247    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6ce7cee-ce2a-4bba-938c-92dde9e042bc-home" (OuterVolumeSpecName: "home") pod "c6ce7cee-ce2a-4bba-938c-92dde9e042bc" (UID: "c6ce7cee-ce2a-4bba-938c-92dde9e042bc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:28:12.547513 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:12.547369    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6ce7cee-ce2a-4bba-938c-92dde9e042bc-model-cache" (OuterVolumeSpecName: "model-cache") pod "c6ce7cee-ce2a-4bba-938c-92dde9e042bc" (UID: "c6ce7cee-ce2a-4bba-938c-92dde9e042bc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:28:12.547513 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:12.547382    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6ce7cee-ce2a-4bba-938c-92dde9e042bc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c6ce7cee-ce2a-4bba-938c-92dde9e042bc" (UID: "c6ce7cee-ce2a-4bba-938c-92dde9e042bc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:28:12.547625 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:12.547545    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6ce7cee-ce2a-4bba-938c-92dde9e042bc-model-cache\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:28:12.547625 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:12.547576    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6ce7cee-ce2a-4bba-938c-92dde9e042bc-tmp-dir\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:28:12.547625 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:12.547585    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6ce7cee-ce2a-4bba-938c-92dde9e042bc-home\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:28:12.549227 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:12.549198    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6ce7cee-ce2a-4bba-938c-92dde9e042bc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c6ce7cee-ce2a-4bba-938c-92dde9e042bc" (UID: "c6ce7cee-ce2a-4bba-938c-92dde9e042bc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:28:12.549669 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:12.549647    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6ce7cee-ce2a-4bba-938c-92dde9e042bc-kube-api-access-mw7rv" (OuterVolumeSpecName: "kube-api-access-mw7rv") pod "c6ce7cee-ce2a-4bba-938c-92dde9e042bc" (UID: "c6ce7cee-ce2a-4bba-938c-92dde9e042bc"). InnerVolumeSpecName "kube-api-access-mw7rv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:28:12.549669 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:12.549659    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6ce7cee-ce2a-4bba-938c-92dde9e042bc-dshm" (OuterVolumeSpecName: "dshm") pod "c6ce7cee-ce2a-4bba-938c-92dde9e042bc" (UID: "c6ce7cee-ce2a-4bba-938c-92dde9e042bc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:28:12.648532 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:12.648508    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6ce7cee-ce2a-4bba-938c-92dde9e042bc-dshm\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:28:12.648532 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:12.648531    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6ce7cee-ce2a-4bba-938c-92dde9e042bc-tls-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:28:12.648681 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:12.648542    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mw7rv\" (UniqueName: \"kubernetes.io/projected/c6ce7cee-ce2a-4bba-938c-92dde9e042bc-kube-api-access-mw7rv\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:28:13.389497 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:13.389445    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b79988/sub-test-model-b79988-kserve-8546cd8769-bhdjb"
Sep 17 14:28:13.389497 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:13.389488    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b79988/sub-test-model-b79988-kserve-8546cd8769-bhdjb" event={"ID":"c6ce7cee-ce2a-4bba-938c-92dde9e042bc","Type":"ContainerDied","Data":"47cd64443054cfd307849c82f0185daf6840e90c23ad521687cd04ee93dcd508"}
Sep 17 14:28:13.390007 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:13.389541    2560 scope.go:117] "RemoveContainer" containerID="da965650654a0fc8e497517b82d693008b5fea961d729f2c8dfe326b8983b216"
Sep 17 14:28:13.412804 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:13.412780    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b79988/sub-test-model-b79988-kserve-8546cd8769-bhdjb"]
Sep 17 14:28:13.418173 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:13.418150    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b79988/sub-test-model-b79988-kserve-8546cd8769-bhdjb"]
Sep 17 14:28:13.572154 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:13.572117    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a057ca78-151c-430c-b15e-15cd4567d9ae" path="/var/lib/kubelet/pods/a057ca78-151c-430c-b15e-15cd4567d9ae/volumes"
Sep 17 14:28:13.572564 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:13.572549    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6ce7cee-ce2a-4bba-938c-92dde9e042bc" path="/var/lib/kubelet/pods/c6ce7cee-ce2a-4bba-938c-92dde9e042bc/volumes"
Sep 17 14:28:34.392025 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:34.391988    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:28:34.492371 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:34.492341    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:28:38.826690 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:38.826659    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-b79988/rate-test-model-b79988-kserve-7bd599d44c-x2xth"]
Sep 17 14:28:38.827055 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:38.827043    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a057ca78-151c-430c-b15e-15cd4567d9ae" containerName="main"
Sep 17 14:28:38.827099 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:38.827056    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a057ca78-151c-430c-b15e-15cd4567d9ae" containerName="main"
Sep 17 14:28:38.827099 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:38.827069    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6ce7cee-ce2a-4bba-938c-92dde9e042bc" containerName="main"
Sep 17 14:28:38.827099 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:38.827075    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6ce7cee-ce2a-4bba-938c-92dde9e042bc" containerName="main"
Sep 17 14:28:38.827198 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:38.827136    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6ce7cee-ce2a-4bba-938c-92dde9e042bc" containerName="main"
Sep 17 14:28:38.827198 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:38.827169    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a057ca78-151c-430c-b15e-15cd4567d9ae" containerName="main"
Sep 17 14:28:38.830135 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:38.830120    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b79988/rate-test-model-b79988-kserve-7bd599d44c-x2xth"
Sep 17 14:28:38.833050 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:38.833028    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-b79988\"/\"openshift-service-ca.crt\""
Sep 17 14:28:38.833050 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:38.833047    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-b79988\"/\"kube-root-ca.crt\""
Sep 17 14:28:38.833204 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:38.833028    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-b79988\"/\"rate-test-model-b79988-kserve-self-signed-certs\""
Sep 17 14:28:38.833204 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:38.833030    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-b79988\"/\"default-dockercfg-gjn9m\""
Sep 17 14:28:38.838760 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:38.838740    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b79988/rate-test-model-b79988-kserve-7bd599d44c-x2xth"]
Sep 17 14:28:38.914276 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:38.914246    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:28:38.963441 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:38.963415    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da8eb37a-aee0-42bd-9e62-c30abab816a0-home\") pod \"rate-test-model-b79988-kserve-7bd599d44c-x2xth\" (UID: \"da8eb37a-aee0-42bd-9e62-c30abab816a0\") " pod="ai-tenant-e2e-shared-b-w3-b79988/rate-test-model-b79988-kserve-7bd599d44c-x2xth"
Sep 17 14:28:38.963584 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:38.963486    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95kxc\" (UniqueName: \"kubernetes.io/projected/da8eb37a-aee0-42bd-9e62-c30abab816a0-kube-api-access-95kxc\") pod \"rate-test-model-b79988-kserve-7bd599d44c-x2xth\" (UID: \"da8eb37a-aee0-42bd-9e62-c30abab816a0\") " pod="ai-tenant-e2e-shared-b-w3-b79988/rate-test-model-b79988-kserve-7bd599d44c-x2xth"
Sep 17 14:28:38.963584 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:38.963513    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da8eb37a-aee0-42bd-9e62-c30abab816a0-dshm\") pod \"rate-test-model-b79988-kserve-7bd599d44c-x2xth\" (UID: \"da8eb37a-aee0-42bd-9e62-c30abab816a0\") " pod="ai-tenant-e2e-shared-b-w3-b79988/rate-test-model-b79988-kserve-7bd599d44c-x2xth"
Sep 17 14:28:38.963584 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:38.963542    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da8eb37a-aee0-42bd-9e62-c30abab816a0-model-cache\") pod \"rate-test-model-b79988-kserve-7bd599d44c-x2xth\" (UID: \"da8eb37a-aee0-42bd-9e62-c30abab816a0\") " pod="ai-tenant-e2e-shared-b-w3-b79988/rate-test-model-b79988-kserve-7bd599d44c-x2xth"
Sep 17 14:28:38.963584 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:38.963580    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da8eb37a-aee0-42bd-9e62-c30abab816a0-tmp-dir\") pod \"rate-test-model-b79988-kserve-7bd599d44c-x2xth\" (UID: \"da8eb37a-aee0-42bd-9e62-c30abab816a0\") " pod="ai-tenant-e2e-shared-b-w3-b79988/rate-test-model-b79988-kserve-7bd599d44c-x2xth"
Sep 17 14:28:38.963719 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:38.963603    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da8eb37a-aee0-42bd-9e62-c30abab816a0-tls-certs\") pod \"rate-test-model-b79988-kserve-7bd599d44c-x2xth\" (UID: \"da8eb37a-aee0-42bd-9e62-c30abab816a0\") " pod="ai-tenant-e2e-shared-b-w3-b79988/rate-test-model-b79988-kserve-7bd599d44c-x2xth"
Sep 17 14:28:39.064315 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:39.064294    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da8eb37a-aee0-42bd-9e62-c30abab816a0-home\") pod \"rate-test-model-b79988-kserve-7bd599d44c-x2xth\" (UID: \"da8eb37a-aee0-42bd-9e62-c30abab816a0\") " pod="ai-tenant-e2e-shared-b-w3-b79988/rate-test-model-b79988-kserve-7bd599d44c-x2xth"
Sep 17 14:28:39.064423 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:39.064334    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95kxc\" (UniqueName: \"kubernetes.io/projected/da8eb37a-aee0-42bd-9e62-c30abab816a0-kube-api-access-95kxc\") pod \"rate-test-model-b79988-kserve-7bd599d44c-x2xth\" (UID: \"da8eb37a-aee0-42bd-9e62-c30abab816a0\") " pod="ai-tenant-e2e-shared-b-w3-b79988/rate-test-model-b79988-kserve-7bd599d44c-x2xth"
Sep 17 14:28:39.064423 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:39.064358    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da8eb37a-aee0-42bd-9e62-c30abab816a0-dshm\") pod \"rate-test-model-b79988-kserve-7bd599d44c-x2xth\" (UID: \"da8eb37a-aee0-42bd-9e62-c30abab816a0\") " pod="ai-tenant-e2e-shared-b-w3-b79988/rate-test-model-b79988-kserve-7bd599d44c-x2xth"
Sep 17 14:28:39.064423 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:39.064382    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da8eb37a-aee0-42bd-9e62-c30abab816a0-model-cache\") pod \"rate-test-model-b79988-kserve-7bd599d44c-x2xth\" (UID: \"da8eb37a-aee0-42bd-9e62-c30abab816a0\") " pod="ai-tenant-e2e-shared-b-w3-b79988/rate-test-model-b79988-kserve-7bd599d44c-x2xth"
Sep 17 14:28:39.064689 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:39.064429    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da8eb37a-aee0-42bd-9e62-c30abab816a0-tmp-dir\") pod \"rate-test-model-b79988-kserve-7bd599d44c-x2xth\" (UID: \"da8eb37a-aee0-42bd-9e62-c30abab816a0\") " pod="ai-tenant-e2e-shared-b-w3-b79988/rate-test-model-b79988-kserve-7bd599d44c-x2xth"
Sep 17 14:28:39.064689 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:39.064481    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da8eb37a-aee0-42bd-9e62-c30abab816a0-tls-certs\") pod \"rate-test-model-b79988-kserve-7bd599d44c-x2xth\" (UID: \"da8eb37a-aee0-42bd-9e62-c30abab816a0\") " pod="ai-tenant-e2e-shared-b-w3-b79988/rate-test-model-b79988-kserve-7bd599d44c-x2xth"
Sep 17 14:28:39.064886 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:39.064867    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da8eb37a-aee0-42bd-9e62-c30abab816a0-model-cache\") pod \"rate-test-model-b79988-kserve-7bd599d44c-x2xth\" (UID: \"da8eb37a-aee0-42bd-9e62-c30abab816a0\") " pod="ai-tenant-e2e-shared-b-w3-b79988/rate-test-model-b79988-kserve-7bd599d44c-x2xth"
Sep 17 14:28:39.064942 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:39.064904    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da8eb37a-aee0-42bd-9e62-c30abab816a0-home\") pod \"rate-test-model-b79988-kserve-7bd599d44c-x2xth\" (UID: \"da8eb37a-aee0-42bd-9e62-c30abab816a0\") " pod="ai-tenant-e2e-shared-b-w3-b79988/rate-test-model-b79988-kserve-7bd599d44c-x2xth"
Sep 17 14:28:39.064942 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:39.064864    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da8eb37a-aee0-42bd-9e62-c30abab816a0-tmp-dir\") pod \"rate-test-model-b79988-kserve-7bd599d44c-x2xth\" (UID: \"da8eb37a-aee0-42bd-9e62-c30abab816a0\") " pod="ai-tenant-e2e-shared-b-w3-b79988/rate-test-model-b79988-kserve-7bd599d44c-x2xth"
Sep 17 14:28:39.066598 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:39.066578    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da8eb37a-aee0-42bd-9e62-c30abab816a0-dshm\") pod \"rate-test-model-b79988-kserve-7bd599d44c-x2xth\" (UID: \"da8eb37a-aee0-42bd-9e62-c30abab816a0\") " pod="ai-tenant-e2e-shared-b-w3-b79988/rate-test-model-b79988-kserve-7bd599d44c-x2xth"
Sep 17 14:28:39.066853 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:39.066836    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da8eb37a-aee0-42bd-9e62-c30abab816a0-tls-certs\") pod \"rate-test-model-b79988-kserve-7bd599d44c-x2xth\" (UID: \"da8eb37a-aee0-42bd-9e62-c30abab816a0\") " pod="ai-tenant-e2e-shared-b-w3-b79988/rate-test-model-b79988-kserve-7bd599d44c-x2xth"
Sep 17 14:28:39.079661 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:39.079616    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95kxc\" (UniqueName: \"kubernetes.io/projected/da8eb37a-aee0-42bd-9e62-c30abab816a0-kube-api-access-95kxc\") pod \"rate-test-model-b79988-kserve-7bd599d44c-x2xth\" (UID: \"da8eb37a-aee0-42bd-9e62-c30abab816a0\") " pod="ai-tenant-e2e-shared-b-w3-b79988/rate-test-model-b79988-kserve-7bd599d44c-x2xth"
Sep 17 14:28:39.140313 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:39.140284    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b79988/rate-test-model-b79988-kserve-7bd599d44c-x2xth"
Sep 17 14:28:39.416417 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:39.415540    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b79988/rate-test-model-b79988-kserve-7bd599d44c-x2xth"]
Sep 17 14:28:39.418764 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:28:39.418737    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda8eb37a_aee0_42bd_9e62_c30abab816a0.slice/crio-c212bf4cf4481438527335d9c5275a99d2d5f720f60f7188bcc52822d318c8be WatchSource:0}: Error finding container c212bf4cf4481438527335d9c5275a99d2d5f720f60f7188bcc52822d318c8be: Status 404 returned error can't find the container with id c212bf4cf4481438527335d9c5275a99d2d5f720f60f7188bcc52822d318c8be
Sep 17 14:28:39.552380 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:39.552347    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b79988/rate-test-model-b79988-kserve-7bd599d44c-x2xth" event={"ID":"da8eb37a-aee0-42bd-9e62-c30abab816a0","Type":"ContainerStarted","Data":"79c39a3b6badeaa3ef56315e6eba866daaeaf2e7024282b7920baa76b00fcc7b"}
Sep 17 14:28:39.552543 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:39.552395    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b79988/rate-test-model-b79988-kserve-7bd599d44c-x2xth" event={"ID":"da8eb37a-aee0-42bd-9e62-c30abab816a0","Type":"ContainerStarted","Data":"c212bf4cf4481438527335d9c5275a99d2d5f720f60f7188bcc52822d318c8be"}
Sep 17 14:28:39.581643 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:39.581601    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-b79988/rate-test-model-b79988-kserve-7bd599d44c-x2xth" podStartSLOduration=1.581587429 podStartE2EDuration="1.581587429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:28:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:28:39.579712947 +0000 UTC m=+1988.521856592" watchObservedRunningTime="2026-09-17 14:28:39.581587429 +0000 UTC m=+1988.523731075"
Sep 17 14:28:49.140595 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:49.140560    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-b79988/rate-test-model-b79988-kserve-7bd599d44c-x2xth"
Sep 17 14:28:49.141065 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:49.140608    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-b79988/rate-test-model-b79988-kserve-7bd599d44c-x2xth"
Sep 17 14:28:49.153036 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:49.153010    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-b79988/rate-test-model-b79988-kserve-7bd599d44c-x2xth"
Sep 17 14:28:49.618911 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:28:49.618876    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-b79988/rate-test-model-b79988-kserve-7bd599d44c-x2xth"
Sep 17 14:29:01.191124 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:01.191090    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:21.828162 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:21.828123    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:22.750788 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:22.750756    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b79988/rate-test-model-b79988-kserve-7bd599d44c-x2xth"]
Sep 17 14:29:22.814436 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:22.814407    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:23.213358 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:23.213328    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:23.620321 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:23.620294    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf"]
Sep 17 14:29:23.623872 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:23.623857    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf"
Sep 17 14:29:23.634045 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:23.634023    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-911a82-openshift-default-dockercfg-kngmk\""
Sep 17 14:29:23.653594 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:23.653573    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf"]
Sep 17 14:29:23.673782 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:23.673752    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:23.752602 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:23.752532    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6hl4\" (UniqueName: \"kubernetes.io/projected/4633a212-1cff-495c-80df-ceaa196237ae-kube-api-access-l6hl4\") pod \"e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf\" (UID: \"4633a212-1cff-495c-80df-ceaa196237ae\") " pod="openshift-ingress/e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf"
Sep 17 14:29:23.752602 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:23.752603    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4633a212-1cff-495c-80df-ceaa196237ae-istiod-ca-cert\") pod \"e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf\" (UID: \"4633a212-1cff-495c-80df-ceaa196237ae\") " pod="openshift-ingress/e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf"
Sep 17 14:29:23.752796 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:23.752649    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4633a212-1cff-495c-80df-ceaa196237ae-istio-podinfo\") pod \"e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf\" (UID: \"4633a212-1cff-495c-80df-ceaa196237ae\") " pod="openshift-ingress/e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf"
Sep 17 14:29:23.752796 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:23.752709    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4633a212-1cff-495c-80df-ceaa196237ae-istio-data\") pod \"e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf\" (UID: \"4633a212-1cff-495c-80df-ceaa196237ae\") " pod="openshift-ingress/e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf"
Sep 17 14:29:23.752796 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:23.752740    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4633a212-1cff-495c-80df-ceaa196237ae-istio-token\") pod \"e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf\" (UID: \"4633a212-1cff-495c-80df-ceaa196237ae\") " pod="openshift-ingress/e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf"
Sep 17 14:29:23.752796 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:23.752765    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4633a212-1cff-495c-80df-ceaa196237ae-credential-socket\") pod \"e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf\" (UID: \"4633a212-1cff-495c-80df-ceaa196237ae\") " pod="openshift-ingress/e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf"
Sep 17 14:29:23.752796 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:23.752783    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4633a212-1cff-495c-80df-ceaa196237ae-workload-certs\") pod \"e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf\" (UID: \"4633a212-1cff-495c-80df-ceaa196237ae\") " pod="openshift-ingress/e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf"
Sep 17 14:29:23.752966 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:23.752801    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4633a212-1cff-495c-80df-ceaa196237ae-istio-envoy\") pod \"e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf\" (UID: \"4633a212-1cff-495c-80df-ceaa196237ae\") " pod="openshift-ingress/e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf"
Sep 17 14:29:23.752966 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:23.752875    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4633a212-1cff-495c-80df-ceaa196237ae-workload-socket\") pod \"e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf\" (UID: \"4633a212-1cff-495c-80df-ceaa196237ae\") " pod="openshift-ingress/e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf"
Sep 17 14:29:23.853642 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:23.853613    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4633a212-1cff-495c-80df-ceaa196237ae-istio-podinfo\") pod \"e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf\" (UID: \"4633a212-1cff-495c-80df-ceaa196237ae\") " pod="openshift-ingress/e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf"
Sep 17 14:29:23.853793 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:23.853653    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4633a212-1cff-495c-80df-ceaa196237ae-istio-data\") pod \"e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf\" (UID: \"4633a212-1cff-495c-80df-ceaa196237ae\") " pod="openshift-ingress/e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf"
Sep 17 14:29:23.853793 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:23.853672    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4633a212-1cff-495c-80df-ceaa196237ae-istio-token\") pod \"e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf\" (UID: \"4633a212-1cff-495c-80df-ceaa196237ae\") " pod="openshift-ingress/e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf"
Sep 17 14:29:23.853793 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:23.853692    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4633a212-1cff-495c-80df-ceaa196237ae-credential-socket\") pod \"e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf\" (UID: \"4633a212-1cff-495c-80df-ceaa196237ae\") " pod="openshift-ingress/e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf"
Sep 17 14:29:23.853793 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:23.853707    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4633a212-1cff-495c-80df-ceaa196237ae-workload-certs\") pod \"e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf\" (UID: \"4633a212-1cff-495c-80df-ceaa196237ae\") " pod="openshift-ingress/e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf"
Sep 17 14:29:23.854011 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:23.853825    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4633a212-1cff-495c-80df-ceaa196237ae-istio-envoy\") pod \"e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf\" (UID: \"4633a212-1cff-495c-80df-ceaa196237ae\") " pod="openshift-ingress/e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf"
Sep 17 14:29:23.854011 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:23.853870    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4633a212-1cff-495c-80df-ceaa196237ae-workload-socket\") pod \"e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf\" (UID: \"4633a212-1cff-495c-80df-ceaa196237ae\") " pod="openshift-ingress/e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf"
Sep 17 14:29:23.854011 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:23.853947    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6hl4\" (UniqueName: \"kubernetes.io/projected/4633a212-1cff-495c-80df-ceaa196237ae-kube-api-access-l6hl4\") pod \"e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf\" (UID: \"4633a212-1cff-495c-80df-ceaa196237ae\") " pod="openshift-ingress/e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf"
Sep 17 14:29:23.854011 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:23.853990    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4633a212-1cff-495c-80df-ceaa196237ae-istiod-ca-cert\") pod \"e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf\" (UID: \"4633a212-1cff-495c-80df-ceaa196237ae\") " pod="openshift-ingress/e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf"
Sep 17 14:29:23.854206 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:23.854060    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4633a212-1cff-495c-80df-ceaa196237ae-istio-data\") pod \"e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf\" (UID: \"4633a212-1cff-495c-80df-ceaa196237ae\") " pod="openshift-ingress/e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf"
Sep 17 14:29:23.854206 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:23.854104    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4633a212-1cff-495c-80df-ceaa196237ae-workload-certs\") pod \"e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf\" (UID: \"4633a212-1cff-495c-80df-ceaa196237ae\") " pod="openshift-ingress/e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf"
Sep 17 14:29:23.854309 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:23.854247    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4633a212-1cff-495c-80df-ceaa196237ae-credential-socket\") pod \"e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf\" (UID: \"4633a212-1cff-495c-80df-ceaa196237ae\") " pod="openshift-ingress/e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf"
Sep 17 14:29:23.854362 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:23.854336    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4633a212-1cff-495c-80df-ceaa196237ae-workload-socket\") pod \"e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf\" (UID: \"4633a212-1cff-495c-80df-ceaa196237ae\") " pod="openshift-ingress/e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf"
Sep 17 14:29:23.854653 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:23.854631    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4633a212-1cff-495c-80df-ceaa196237ae-istiod-ca-cert\") pod \"e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf\" (UID: \"4633a212-1cff-495c-80df-ceaa196237ae\") " pod="openshift-ingress/e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf"
Sep 17 14:29:23.856113 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:23.856090    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4633a212-1cff-495c-80df-ceaa196237ae-istio-envoy\") pod \"e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf\" (UID: \"4633a212-1cff-495c-80df-ceaa196237ae\") " pod="openshift-ingress/e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf"
Sep 17 14:29:23.856223 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:23.856170    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4633a212-1cff-495c-80df-ceaa196237ae-istio-podinfo\") pod \"e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf\" (UID: \"4633a212-1cff-495c-80df-ceaa196237ae\") " pod="openshift-ingress/e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf"
Sep 17 14:29:23.862001 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:23.861980    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4633a212-1cff-495c-80df-ceaa196237ae-istio-token\") pod \"e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf\" (UID: \"4633a212-1cff-495c-80df-ceaa196237ae\") " pod="openshift-ingress/e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf"
Sep 17 14:29:23.862408 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:23.862389    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6hl4\" (UniqueName: \"kubernetes.io/projected/4633a212-1cff-495c-80df-ceaa196237ae-kube-api-access-l6hl4\") pod \"e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf\" (UID: \"4633a212-1cff-495c-80df-ceaa196237ae\") " pod="openshift-ingress/e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf"
Sep 17 14:29:23.934735 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:23.934670    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf"
Sep 17 14:29:24.149052 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:24.149016    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:24.217202 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:24.217170    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf"]
Sep 17 14:29:24.218410 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:29:24.218381    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4633a212_1cff_495c_80df_ceaa196237ae.slice/crio-cc3b4b9f6fe813bed0d3b5a79ebac354bb31f919b215f1cfe93129888eada71b WatchSource:0}: Error finding container cc3b4b9f6fe813bed0d3b5a79ebac354bb31f919b215f1cfe93129888eada71b: Status 404 returned error can't find the container with id cc3b4b9f6fe813bed0d3b5a79ebac354bb31f919b215f1cfe93129888eada71b
Sep 17 14:29:24.220711 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:24.220682    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:29:24.220781 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:24.220754    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:29:24.220822 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:24.220784    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:29:24.812302 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:24.812266    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf" event={"ID":"4633a212-1cff-495c-80df-ceaa196237ae","Type":"ContainerStarted","Data":"bd8ac9dafdf589a3968e1c43dc6f231cf6ac0cc77fdb54e92cd1de2dc998ddf4"}
Sep 17 14:29:24.812545 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:24.812332    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf" event={"ID":"4633a212-1cff-495c-80df-ceaa196237ae","Type":"ContainerStarted","Data":"cc3b4b9f6fe813bed0d3b5a79ebac354bb31f919b215f1cfe93129888eada71b"}
Sep 17 14:29:24.878679 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:24.878626    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf" podStartSLOduration=1.878609263 podStartE2EDuration="1.878609263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:29:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:29:24.869053222 +0000 UTC m=+2033.811196889" watchObservedRunningTime="2026-09-17 14:29:24.878609263 +0000 UTC m=+2033.820752908"
Sep 17 14:29:24.935428 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:24.935398    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf"
Sep 17 14:29:25.940413 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:25.940376    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf"
Sep 17 14:29:26.823897 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:26.823862    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf"
Sep 17 14:29:26.824836 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:26.824813    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf"
Sep 17 14:29:26.901853 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:26.901814    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:27.002771 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:27.002725    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:27.102326 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:27.102244    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:28.071477 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:28.071426    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:28.869321 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:28.869286    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:29.039618 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:29.039567    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:29.160864 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:29.160780    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:29.400786 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:29.400751    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:29.500922 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:29.500840    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:37.770264 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:37.770188    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-b79988/rate-test-model-b79988-kserve-7bd599d44c-x2xth" podUID="da8eb37a-aee0-42bd-9e62-c30abab816a0" containerName="main" containerID="cri-o://79c39a3b6badeaa3ef56315e6eba866daaeaf2e7024282b7920baa76b00fcc7b" gracePeriod=45
Sep 17 14:29:37.900998 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:37.900963    2560 generic.go:356] "Generic (PLEG): container finished" podID="da8eb37a-aee0-42bd-9e62-c30abab816a0" containerID="79c39a3b6badeaa3ef56315e6eba866daaeaf2e7024282b7920baa76b00fcc7b" exitCode=0
Sep 17 14:29:37.901104 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:37.900986    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b79988/rate-test-model-b79988-kserve-7bd599d44c-x2xth" event={"ID":"da8eb37a-aee0-42bd-9e62-c30abab816a0","Type":"ContainerDied","Data":"79c39a3b6badeaa3ef56315e6eba866daaeaf2e7024282b7920baa76b00fcc7b"}
Sep 17 14:29:38.034341 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:38.034281    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b79988/rate-test-model-b79988-kserve-7bd599d44c-x2xth"
Sep 17 14:29:38.196596 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:38.196568    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da8eb37a-aee0-42bd-9e62-c30abab816a0-model-cache\") pod \"da8eb37a-aee0-42bd-9e62-c30abab816a0\" (UID: \"da8eb37a-aee0-42bd-9e62-c30abab816a0\") "
Sep 17 14:29:38.196771 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:38.196621    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da8eb37a-aee0-42bd-9e62-c30abab816a0-tls-certs\") pod \"da8eb37a-aee0-42bd-9e62-c30abab816a0\" (UID: \"da8eb37a-aee0-42bd-9e62-c30abab816a0\") "
Sep 17 14:29:38.196771 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:38.196675    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da8eb37a-aee0-42bd-9e62-c30abab816a0-tmp-dir\") pod \"da8eb37a-aee0-42bd-9e62-c30abab816a0\" (UID: \"da8eb37a-aee0-42bd-9e62-c30abab816a0\") "
Sep 17 14:29:38.196771 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:38.196706    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da8eb37a-aee0-42bd-9e62-c30abab816a0-home\") pod \"da8eb37a-aee0-42bd-9e62-c30abab816a0\" (UID: \"da8eb37a-aee0-42bd-9e62-c30abab816a0\") "
Sep 17 14:29:38.196952 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:38.196778    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da8eb37a-aee0-42bd-9e62-c30abab816a0-dshm\") pod \"da8eb37a-aee0-42bd-9e62-c30abab816a0\" (UID: \"da8eb37a-aee0-42bd-9e62-c30abab816a0\") "
Sep 17 14:29:38.196952 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:38.196790    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da8eb37a-aee0-42bd-9e62-c30abab816a0-model-cache" (OuterVolumeSpecName: "model-cache") pod "da8eb37a-aee0-42bd-9e62-c30abab816a0" (UID: "da8eb37a-aee0-42bd-9e62-c30abab816a0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:29:38.196952 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:38.196814    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-95kxc\" (UniqueName: \"kubernetes.io/projected/da8eb37a-aee0-42bd-9e62-c30abab816a0-kube-api-access-95kxc\") pod \"da8eb37a-aee0-42bd-9e62-c30abab816a0\" (UID: \"da8eb37a-aee0-42bd-9e62-c30abab816a0\") "
Sep 17 14:29:38.196952 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:38.196871    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da8eb37a-aee0-42bd-9e62-c30abab816a0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "da8eb37a-aee0-42bd-9e62-c30abab816a0" (UID: "da8eb37a-aee0-42bd-9e62-c30abab816a0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:29:38.197198 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:38.196961    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da8eb37a-aee0-42bd-9e62-c30abab816a0-home" (OuterVolumeSpecName: "home") pod "da8eb37a-aee0-42bd-9e62-c30abab816a0" (UID: "da8eb37a-aee0-42bd-9e62-c30abab816a0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:29:38.197198 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:38.197164    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da8eb37a-aee0-42bd-9e62-c30abab816a0-model-cache\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:29:38.197284 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:38.197207    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da8eb37a-aee0-42bd-9e62-c30abab816a0-tmp-dir\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:29:38.197284 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:38.197222    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da8eb37a-aee0-42bd-9e62-c30abab816a0-home\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:29:38.198742 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:38.198721    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da8eb37a-aee0-42bd-9e62-c30abab816a0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "da8eb37a-aee0-42bd-9e62-c30abab816a0" (UID: "da8eb37a-aee0-42bd-9e62-c30abab816a0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:29:38.198818 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:38.198758    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da8eb37a-aee0-42bd-9e62-c30abab816a0-dshm" (OuterVolumeSpecName: "dshm") pod "da8eb37a-aee0-42bd-9e62-c30abab816a0" (UID: "da8eb37a-aee0-42bd-9e62-c30abab816a0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:29:38.198854 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:38.198820    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da8eb37a-aee0-42bd-9e62-c30abab816a0-kube-api-access-95kxc" (OuterVolumeSpecName: "kube-api-access-95kxc") pod "da8eb37a-aee0-42bd-9e62-c30abab816a0" (UID: "da8eb37a-aee0-42bd-9e62-c30abab816a0"). InnerVolumeSpecName "kube-api-access-95kxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:29:38.298138 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:38.298062    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da8eb37a-aee0-42bd-9e62-c30abab816a0-dshm\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:29:38.298138 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:38.298090    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95kxc\" (UniqueName: \"kubernetes.io/projected/da8eb37a-aee0-42bd-9e62-c30abab816a0-kube-api-access-95kxc\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:29:38.298138 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:38.298101    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da8eb37a-aee0-42bd-9e62-c30abab816a0-tls-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:29:38.909120 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:38.909085    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b79988/rate-test-model-b79988-kserve-7bd599d44c-x2xth"
Sep 17 14:29:38.909120 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:38.909098    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b79988/rate-test-model-b79988-kserve-7bd599d44c-x2xth" event={"ID":"da8eb37a-aee0-42bd-9e62-c30abab816a0","Type":"ContainerDied","Data":"c212bf4cf4481438527335d9c5275a99d2d5f720f60f7188bcc52822d318c8be"}
Sep 17 14:29:38.909627 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:38.909151    2560 scope.go:117] "RemoveContainer" containerID="79c39a3b6badeaa3ef56315e6eba866daaeaf2e7024282b7920baa76b00fcc7b"
Sep 17 14:29:38.934731 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:38.934707    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b79988/rate-test-model-b79988-kserve-7bd599d44c-x2xth"]
Sep 17 14:29:38.939107 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:38.939084    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b79988/rate-test-model-b79988-kserve-7bd599d44c-x2xth"]
Sep 17 14:29:39.570376 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:39.570333    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da8eb37a-aee0-42bd-9e62-c30abab816a0" path="/var/lib/kubelet/pods/da8eb37a-aee0-42bd-9e62-c30abab816a0/volumes"
Sep 17 14:29:41.922030 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:41.921322    2560 patch_prober.go:28] interesting pod/e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": read tcp 10.132.0.2:36274->10.132.0.109:15021: read: connection reset by peer" start-of-body=
Sep 17 14:29:41.922030 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:41.921423    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf" podUID="4633a212-1cff-495c-80df-ceaa196237ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": read tcp 10.132.0.2:36274->10.132.0.109:15021: read: connection reset by peer"
Sep 17 14:29:42.937445 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:42.937411    2560 generic.go:356] "Generic (PLEG): container finished" podID="4633a212-1cff-495c-80df-ceaa196237ae" containerID="bd8ac9dafdf589a3968e1c43dc6f231cf6ac0cc77fdb54e92cd1de2dc998ddf4" exitCode=0
Sep 17 14:29:42.937832 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:42.937499    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf" event={"ID":"4633a212-1cff-495c-80df-ceaa196237ae","Type":"ContainerDied","Data":"bd8ac9dafdf589a3968e1c43dc6f231cf6ac0cc77fdb54e92cd1de2dc998ddf4"}
Sep 17 14:29:42.937915 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:42.937902    2560 scope.go:117] "RemoveContainer" containerID="bd8ac9dafdf589a3968e1c43dc6f231cf6ac0cc77fdb54e92cd1de2dc998ddf4"
Sep 17 14:29:42.938956 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:42.938929    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:29:42.939057 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:42.939006    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:29:42.939057 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:42.939052    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:29:43.945297 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:43.945264    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf" event={"ID":"4633a212-1cff-495c-80df-ceaa196237ae","Type":"ContainerStarted","Data":"9a9723be0ff9a613fabb68ecebcd627f3e5e896183b0d5e02bc4a601b982ab43"}
Sep 17 14:29:44.935547 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:44.935509    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf"
Sep 17 14:29:44.935725 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:44.935699    2560 patch_prober.go:28] interesting pod/e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Sep 17 14:29:44.935772 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:44.935740    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf" podUID="4633a212-1cff-495c-80df-ceaa196237ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Sep 17 14:29:45.935571 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:45.935537    2560 patch_prober.go:28] interesting pod/e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Sep 17 14:29:45.935955 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:45.935590    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf" podUID="4633a212-1cff-495c-80df-ceaa196237ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Sep 17 14:29:46.935556 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:46.935519    2560 patch_prober.go:28] interesting pod/e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Sep 17 14:29:46.935727 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:46.935581    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf" podUID="4633a212-1cff-495c-80df-ceaa196237ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Sep 17 14:29:47.935333 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:47.935299    2560 patch_prober.go:28] interesting pod/e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Sep 17 14:29:47.935538 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:47.935350    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf" podUID="4633a212-1cff-495c-80df-ceaa196237ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Sep 17 14:29:48.935883 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:48.935846    2560 patch_prober.go:28] interesting pod/e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Sep 17 14:29:48.936294 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:48.935907    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf" podUID="4633a212-1cff-495c-80df-ceaa196237ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Sep 17 14:29:49.938308 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:49.938281    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf"
Sep 17 14:29:49.938749 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:49.938734    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf"
Sep 17 14:29:49.939344 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:49.939325    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf"
Sep 17 14:29:52.584287 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:52.584249    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:53.617435 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:53.617408    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:54.369642 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:54.369601    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:55.264385 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:55.264344    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:57.030135 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:57.030098    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:57.766624 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:57.766592    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:58.063387 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:58.063356    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg"]
Sep 17 14:29:58.063814 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:58.063800    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da8eb37a-aee0-42bd-9e62-c30abab816a0" containerName="main"
Sep 17 14:29:58.063814 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:58.063815    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="da8eb37a-aee0-42bd-9e62-c30abab816a0" containerName="main"
Sep 17 14:29:58.063899 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:58.063890    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="da8eb37a-aee0-42bd-9e62-c30abab816a0" containerName="main"
Sep 17 14:29:58.068148 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:58.068131    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg"
Sep 17 14:29:58.070279 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:58.070262    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-166f16-openshift-default-dockercfg-pstsr\""
Sep 17 14:29:58.079116 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:58.079089    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg"]
Sep 17 14:29:58.095608 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:58.095574    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:58.191041 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:58.191012    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7c163b0-e401-4279-aaa2-2def6ac2f74b-istio-podinfo\") pod \"e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg\" (UID: \"b7c163b0-e401-4279-aaa2-2def6ac2f74b\") " pod="openshift-ingress/e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg"
Sep 17 14:29:58.191041 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:58.191043    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7c163b0-e401-4279-aaa2-2def6ac2f74b-istiod-ca-cert\") pod \"e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg\" (UID: \"b7c163b0-e401-4279-aaa2-2def6ac2f74b\") " pod="openshift-ingress/e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg"
Sep 17 14:29:58.191239 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:58.191062    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7c163b0-e401-4279-aaa2-2def6ac2f74b-istio-data\") pod \"e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg\" (UID: \"b7c163b0-e401-4279-aaa2-2def6ac2f74b\") " pod="openshift-ingress/e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg"
Sep 17 14:29:58.191239 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:58.191099    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7c163b0-e401-4279-aaa2-2def6ac2f74b-workload-socket\") pod \"e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg\" (UID: \"b7c163b0-e401-4279-aaa2-2def6ac2f74b\") " pod="openshift-ingress/e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg"
Sep 17 14:29:58.191239 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:58.191176    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7c163b0-e401-4279-aaa2-2def6ac2f74b-workload-certs\") pod \"e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg\" (UID: \"b7c163b0-e401-4279-aaa2-2def6ac2f74b\") " pod="openshift-ingress/e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg"
Sep 17 14:29:58.191239 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:58.191202    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7c163b0-e401-4279-aaa2-2def6ac2f74b-istio-envoy\") pod \"e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg\" (UID: \"b7c163b0-e401-4279-aaa2-2def6ac2f74b\") " pod="openshift-ingress/e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg"
Sep 17 14:29:58.191239 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:58.191236    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7c163b0-e401-4279-aaa2-2def6ac2f74b-istio-token\") pod \"e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg\" (UID: \"b7c163b0-e401-4279-aaa2-2def6ac2f74b\") " pod="openshift-ingress/e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg"
Sep 17 14:29:58.191416 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:58.191297    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7c163b0-e401-4279-aaa2-2def6ac2f74b-credential-socket\") pod \"e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg\" (UID: \"b7c163b0-e401-4279-aaa2-2def6ac2f74b\") " pod="openshift-ingress/e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg"
Sep 17 14:29:58.191416 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:58.191316    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zr2m\" (UniqueName: \"kubernetes.io/projected/b7c163b0-e401-4279-aaa2-2def6ac2f74b-kube-api-access-7zr2m\") pod \"e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg\" (UID: \"b7c163b0-e401-4279-aaa2-2def6ac2f74b\") " pod="openshift-ingress/e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg"
Sep 17 14:29:58.292660 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:58.292617    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7c163b0-e401-4279-aaa2-2def6ac2f74b-workload-certs\") pod \"e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg\" (UID: \"b7c163b0-e401-4279-aaa2-2def6ac2f74b\") " pod="openshift-ingress/e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg"
Sep 17 14:29:58.292660 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:58.292660    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7c163b0-e401-4279-aaa2-2def6ac2f74b-istio-envoy\") pod \"e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg\" (UID: \"b7c163b0-e401-4279-aaa2-2def6ac2f74b\") " pod="openshift-ingress/e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg"
Sep 17 14:29:58.292920 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:58.292692    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7c163b0-e401-4279-aaa2-2def6ac2f74b-istio-token\") pod \"e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg\" (UID: \"b7c163b0-e401-4279-aaa2-2def6ac2f74b\") " pod="openshift-ingress/e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg"
Sep 17 14:29:58.292920 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:58.292717    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7c163b0-e401-4279-aaa2-2def6ac2f74b-credential-socket\") pod \"e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg\" (UID: \"b7c163b0-e401-4279-aaa2-2def6ac2f74b\") " pod="openshift-ingress/e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg"
Sep 17 14:29:58.292920 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:58.292732    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zr2m\" (UniqueName: \"kubernetes.io/projected/b7c163b0-e401-4279-aaa2-2def6ac2f74b-kube-api-access-7zr2m\") pod \"e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg\" (UID: \"b7c163b0-e401-4279-aaa2-2def6ac2f74b\") " pod="openshift-ingress/e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg"
Sep 17 14:29:58.292920 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:58.292771    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7c163b0-e401-4279-aaa2-2def6ac2f74b-istio-podinfo\") pod \"e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg\" (UID: \"b7c163b0-e401-4279-aaa2-2def6ac2f74b\") " pod="openshift-ingress/e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg"
Sep 17 14:29:58.292920 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:58.292795    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7c163b0-e401-4279-aaa2-2def6ac2f74b-istiod-ca-cert\") pod \"e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg\" (UID: \"b7c163b0-e401-4279-aaa2-2def6ac2f74b\") " pod="openshift-ingress/e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg"
Sep 17 14:29:58.293220 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:58.292941    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7c163b0-e401-4279-aaa2-2def6ac2f74b-istio-data\") pod \"e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg\" (UID: \"b7c163b0-e401-4279-aaa2-2def6ac2f74b\") " pod="openshift-ingress/e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg"
Sep 17 14:29:58.293220 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:58.292996    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7c163b0-e401-4279-aaa2-2def6ac2f74b-workload-socket\") pod \"e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg\" (UID: \"b7c163b0-e401-4279-aaa2-2def6ac2f74b\") " pod="openshift-ingress/e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg"
Sep 17 14:29:58.293220 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:58.293079    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7c163b0-e401-4279-aaa2-2def6ac2f74b-workload-certs\") pod \"e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg\" (UID: \"b7c163b0-e401-4279-aaa2-2def6ac2f74b\") " pod="openshift-ingress/e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg"
Sep 17 14:29:58.293220 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:58.293120    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7c163b0-e401-4279-aaa2-2def6ac2f74b-credential-socket\") pod \"e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg\" (UID: \"b7c163b0-e401-4279-aaa2-2def6ac2f74b\") " pod="openshift-ingress/e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg"
Sep 17 14:29:58.293448 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:58.293301    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7c163b0-e401-4279-aaa2-2def6ac2f74b-istio-data\") pod \"e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg\" (UID: \"b7c163b0-e401-4279-aaa2-2def6ac2f74b\") " pod="openshift-ingress/e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg"
Sep 17 14:29:58.293448 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:58.293378    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7c163b0-e401-4279-aaa2-2def6ac2f74b-workload-socket\") pod \"e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg\" (UID: \"b7c163b0-e401-4279-aaa2-2def6ac2f74b\") " pod="openshift-ingress/e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg"
Sep 17 14:29:58.293567 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:58.293546    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7c163b0-e401-4279-aaa2-2def6ac2f74b-istiod-ca-cert\") pod \"e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg\" (UID: \"b7c163b0-e401-4279-aaa2-2def6ac2f74b\") " pod="openshift-ingress/e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg"
Sep 17 14:29:58.294977 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:58.294958    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7c163b0-e401-4279-aaa2-2def6ac2f74b-istio-envoy\") pod \"e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg\" (UID: \"b7c163b0-e401-4279-aaa2-2def6ac2f74b\") " pod="openshift-ingress/e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg"
Sep 17 14:29:58.295268 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:58.295250    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7c163b0-e401-4279-aaa2-2def6ac2f74b-istio-podinfo\") pod \"e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg\" (UID: \"b7c163b0-e401-4279-aaa2-2def6ac2f74b\") " pod="openshift-ingress/e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg"
Sep 17 14:29:58.300636 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:58.300608    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7c163b0-e401-4279-aaa2-2def6ac2f74b-istio-token\") pod \"e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg\" (UID: \"b7c163b0-e401-4279-aaa2-2def6ac2f74b\") " pod="openshift-ingress/e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg"
Sep 17 14:29:58.301532 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:58.301513    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zr2m\" (UniqueName: \"kubernetes.io/projected/b7c163b0-e401-4279-aaa2-2def6ac2f74b-kube-api-access-7zr2m\") pod \"e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg\" (UID: \"b7c163b0-e401-4279-aaa2-2def6ac2f74b\") " pod="openshift-ingress/e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg"
Sep 17 14:29:58.379872 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:58.379791    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg"
Sep 17 14:29:58.656251 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:58.656218    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg"]
Sep 17 14:29:58.657833 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:29:58.657809    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7c163b0_e401_4279_aaa2_2def6ac2f74b.slice/crio-d266440425bd4122da2cdf320eba6c7a3809f5b391c4a4fc47b3c69e34bb181a WatchSource:0}: Error finding container d266440425bd4122da2cdf320eba6c7a3809f5b391c4a4fc47b3c69e34bb181a: Status 404 returned error can't find the container with id d266440425bd4122da2cdf320eba6c7a3809f5b391c4a4fc47b3c69e34bb181a
Sep 17 14:29:58.660292 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:58.660254    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:29:58.660361 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:58.660342    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:29:58.660405 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:58.660389    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:29:58.728110 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:58.728075    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:59.050711 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:59.050682    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg" event={"ID":"b7c163b0-e401-4279-aaa2-2def6ac2f74b","Type":"ContainerStarted","Data":"27bbfc02491f40cd2ed63dc321ce125db24e4212360a123e18b23da49d32d4b1"}
Sep 17 14:29:59.050876 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:59.050732    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg" event={"ID":"b7c163b0-e401-4279-aaa2-2def6ac2f74b","Type":"ContainerStarted","Data":"d266440425bd4122da2cdf320eba6c7a3809f5b391c4a4fc47b3c69e34bb181a"}
Sep 17 14:29:59.072795 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:59.072747    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg" podStartSLOduration=1.072733547 podStartE2EDuration="1.072733547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:29:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:29:59.069747727 +0000 UTC m=+2068.011891371" watchObservedRunningTime="2026-09-17 14:29:59.072733547 +0000 UTC m=+2068.014877193"
Sep 17 14:29:59.380184 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:59.380103    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg"
Sep 17 14:29:59.381521 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:59.381497    2560 patch_prober.go:28] interesting pod/e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Sep 17 14:29:59.381590 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:29:59.381548    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg" podUID="b7c163b0-e401-4279-aaa2-2def6ac2f74b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Sep 17 14:30:00.145448 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:00.145414    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29827590-p8rkr"]
Sep 17 14:30:00.149785 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:00.149761    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b88b90-29827590-ngjln"]
Sep 17 14:30:00.149903 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:00.149859    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29827590-p8rkr"
Sep 17 14:30:00.152068 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:00.152052    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rp9ww\""
Sep 17 14:30:00.153308 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:00.153286    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b88b90-29827590-ngjln"
Sep 17 14:30:00.160680 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:00.160656    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29827590-p8rkr"]
Sep 17 14:30:00.164228 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:00.164185    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b88b90-29827590-ngjln"]
Sep 17 14:30:00.164490 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:00.164447    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29827590-p8rkr"
Sep 17 14:30:00.172255 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:00.172230    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b88b90-29827590-ngjln"
Sep 17 14:30:00.384974 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:00.384940    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg"
Sep 17 14:30:00.457251 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:00.457212    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29827590-p8rkr"]
Sep 17 14:30:00.458211 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:30:00.458185    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4896d33f_f407_4db7_9473_75db5411ffb5.slice/crio-faeedb705485984a09c8e1d5df5c8e2ca1c352c0667f06f06afeacf37371e854 WatchSource:0}: Error finding container faeedb705485984a09c8e1d5df5c8e2ca1c352c0667f06f06afeacf37371e854: Status 404 returned error can't find the container with id faeedb705485984a09c8e1d5df5c8e2ca1c352c0667f06f06afeacf37371e854
Sep 17 14:30:00.460004 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:00.459985    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b88b90-29827590-ngjln"]
Sep 17 14:30:00.460400 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:30:00.460377    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94f0a0b6_9286_421c_8c15_c382f63dbef2.slice/crio-bdeb3ed5c5e278ee78f9292728766e680a1846b16d6fdf8c042de956bc940c5d WatchSource:0}: Error finding container bdeb3ed5c5e278ee78f9292728766e680a1846b16d6fdf8c042de956bc940c5d: Status 404 returned error can't find the container with id bdeb3ed5c5e278ee78f9292728766e680a1846b16d6fdf8c042de956bc940c5d
Sep 17 14:30:01.061985 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:01.061950    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:30:01.067338 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:01.067281    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29827590-p8rkr" event={"ID":"4896d33f-f407-4db7-9473-75db5411ffb5","Type":"ContainerStarted","Data":"a0697809bbf5e6bdfae7bf92f9958c7b22e7c23fd40a8b27f4c36634565e7957"}
Sep 17 14:30:01.067536 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:01.067344    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29827590-p8rkr" event={"ID":"4896d33f-f407-4db7-9473-75db5411ffb5","Type":"ContainerStarted","Data":"faeedb705485984a09c8e1d5df5c8e2ca1c352c0667f06f06afeacf37371e854"}
Sep 17 14:30:01.069238 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:01.069213    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b88b90-29827590-ngjln" event={"ID":"94f0a0b6-9286-421c-8c15-c382f63dbef2","Type":"ContainerStarted","Data":"91860f2edd354e4cb975af254ee49f900d7689c42c3f447782a1cd8212bb7cbc"}
Sep 17 14:30:01.069346 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:01.069261    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b88b90-29827590-ngjln" event={"ID":"94f0a0b6-9286-421c-8c15-c382f63dbef2","Type":"ContainerStarted","Data":"bdeb3ed5c5e278ee78f9292728766e680a1846b16d6fdf8c042de956bc940c5d"}
Sep 17 14:30:01.069479 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:01.069448    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg"
Sep 17 14:30:01.070450 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:01.070434    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg"
Sep 17 14:30:01.100607 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:01.100556    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29827590-p8rkr" podStartSLOduration=1.100542045 podStartE2EDuration="1.100542045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:30:01.09799206 +0000 UTC m=+2070.040135705" watchObservedRunningTime="2026-09-17 14:30:01.100542045 +0000 UTC m=+2070.042685690"
Sep 17 14:30:01.142024 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:01.141974    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b88b90-29827590-ngjln" podStartSLOduration=1.14195821 podStartE2EDuration="1.14195821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:30:01.140070157 +0000 UTC m=+2070.082213802" watchObservedRunningTime="2026-09-17 14:30:01.14195821 +0000 UTC m=+2070.084101853"
Sep 17 14:30:01.186550 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:01.186516    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:30:01.309655 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:01.309616    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:30:02.077011 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:02.076978    2560 generic.go:356] "Generic (PLEG): container finished" podID="94f0a0b6-9286-421c-8c15-c382f63dbef2" containerID="91860f2edd354e4cb975af254ee49f900d7689c42c3f447782a1cd8212bb7cbc" exitCode=0
Sep 17 14:30:02.077186 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:02.077066    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b88b90-29827590-ngjln" event={"ID":"94f0a0b6-9286-421c-8c15-c382f63dbef2","Type":"ContainerDied","Data":"91860f2edd354e4cb975af254ee49f900d7689c42c3f447782a1cd8212bb7cbc"}
Sep 17 14:30:02.078794 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:02.078773    2560 generic.go:356] "Generic (PLEG): container finished" podID="4896d33f-f407-4db7-9473-75db5411ffb5" containerID="a0697809bbf5e6bdfae7bf92f9958c7b22e7c23fd40a8b27f4c36634565e7957" exitCode=0
Sep 17 14:30:02.078931 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:02.078854    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29827590-p8rkr" event={"ID":"4896d33f-f407-4db7-9473-75db5411ffb5","Type":"ContainerDied","Data":"a0697809bbf5e6bdfae7bf92f9958c7b22e7c23fd40a8b27f4c36634565e7957"}
Sep 17 14:30:02.673364 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:02.673330    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:30:03.241232 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:03.241206    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29827590-p8rkr"
Sep 17 14:30:03.244354 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:03.244334    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b88b90-29827590-ngjln"
Sep 17 14:30:03.882620 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:03.882537    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:30:04.095826 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:04.095798    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29827590-p8rkr"
Sep 17 14:30:04.095984 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:04.095842    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29827590-p8rkr" event={"ID":"4896d33f-f407-4db7-9473-75db5411ffb5","Type":"ContainerDied","Data":"faeedb705485984a09c8e1d5df5c8e2ca1c352c0667f06f06afeacf37371e854"}
Sep 17 14:30:04.095984 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:04.095889    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="faeedb705485984a09c8e1d5df5c8e2ca1c352c0667f06f06afeacf37371e854"
Sep 17 14:30:04.097303 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:04.097281    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b88b90-29827590-ngjln" event={"ID":"94f0a0b6-9286-421c-8c15-c382f63dbef2","Type":"ContainerDied","Data":"bdeb3ed5c5e278ee78f9292728766e680a1846b16d6fdf8c042de956bc940c5d"}
Sep 17 14:30:04.097303 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:04.097303    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b88b90-29827590-ngjln"
Sep 17 14:30:04.097525 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:04.097315    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bdeb3ed5c5e278ee78f9292728766e680a1846b16d6fdf8c042de956bc940c5d"
Sep 17 14:30:04.928917 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:04.928874    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:30:05.025955 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:05.025921    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:30:05.126427 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:05.126396    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:30:05.506163 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:05.506134    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:30:05.612569 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:05.612532    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:30:17.111661 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:17.111614    2560 patch_prober.go:28] interesting pod/e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 17 14:30:17.112056 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:17.111698    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg" podUID="b7c163b0-e401-4279-aaa2-2def6ac2f74b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 17 14:30:19.198830 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:19.198796    2560 generic.go:356] "Generic (PLEG): container finished" podID="b7c163b0-e401-4279-aaa2-2def6ac2f74b" containerID="27bbfc02491f40cd2ed63dc321ce125db24e4212360a123e18b23da49d32d4b1" exitCode=0
Sep 17 14:30:19.199237 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:19.198862    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg" event={"ID":"b7c163b0-e401-4279-aaa2-2def6ac2f74b","Type":"ContainerDied","Data":"27bbfc02491f40cd2ed63dc321ce125db24e4212360a123e18b23da49d32d4b1"}
Sep 17 14:30:19.199290 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:19.199245    2560 scope.go:117] "RemoveContainer" containerID="27bbfc02491f40cd2ed63dc321ce125db24e4212360a123e18b23da49d32d4b1"
Sep 17 14:30:19.200384 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:19.200356    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:30:19.200450 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:19.200428    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:30:19.200526 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:19.200474    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:30:20.208149 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:20.208118    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg" event={"ID":"b7c163b0-e401-4279-aaa2-2def6ac2f74b","Type":"ContainerStarted","Data":"5c988c9123e5357e7bc715427e2e95d977e6ad2abfc0865ee0d2d4c19735eb4f"}
Sep 17 14:30:20.380540 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:20.380498    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg"
Sep 17 14:30:20.380738 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:20.380714    2560 patch_prober.go:28] interesting pod/e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Sep 17 14:30:20.380789 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:20.380762    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg" podUID="b7c163b0-e401-4279-aaa2-2def6ac2f74b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Sep 17 14:30:21.381021 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:21.380978    2560 patch_prober.go:28] interesting pod/e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Sep 17 14:30:21.381554 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:21.381051    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg" podUID="b7c163b0-e401-4279-aaa2-2def6ac2f74b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Sep 17 14:30:22.381099 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:22.381062    2560 patch_prober.go:28] interesting pod/e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Sep 17 14:30:22.381504 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:22.381120    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg" podUID="b7c163b0-e401-4279-aaa2-2def6ac2f74b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Sep 17 14:30:22.885127 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:22.885098    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:30:23.380564 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:23.380522    2560 patch_prober.go:28] interesting pod/e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Sep 17 14:30:23.380746 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:23.380588    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg" podUID="b7c163b0-e401-4279-aaa2-2def6ac2f74b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Sep 17 14:30:24.380511 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:24.380443    2560 patch_prober.go:28] interesting pod/e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Sep 17 14:30:24.380938 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:24.380547    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg" podUID="b7c163b0-e401-4279-aaa2-2def6ac2f74b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Sep 17 14:30:25.407419 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:25.407385    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg"
Sep 17 14:30:25.407885 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:25.407756    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg"
Sep 17 14:30:25.408604 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:25.408578    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg"
Sep 17 14:30:27.283473 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:27.283400    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:30:28.937153 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:28.937119    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:30:29.811965 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:29.811927    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:30:30.760408 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:30.760366    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:30:31.331134 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:31.331102    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:30:31.550309 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:31.550261    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:30:32.590820 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:32.590781    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:30:49.113391 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:49.113361    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-911a82/ipp-route-911a82-kserve-78fb7c6647-zccjm"]
Sep 17 14:30:49.113860 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:49.113837    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94f0a0b6-9286-421c-8c15-c382f63dbef2" containerName="cleanup"
Sep 17 14:30:49.113860 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:49.113850    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="94f0a0b6-9286-421c-8c15-c382f63dbef2" containerName="cleanup"
Sep 17 14:30:49.113934 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:49.113862    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4896d33f-f407-4db7-9473-75db5411ffb5" containerName="cleanup"
Sep 17 14:30:49.113934 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:49.113868    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4896d33f-f407-4db7-9473-75db5411ffb5" containerName="cleanup"
Sep 17 14:30:49.113997 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:49.113949    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="94f0a0b6-9286-421c-8c15-c382f63dbef2" containerName="cleanup"
Sep 17 14:30:49.113997 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:49.113960    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4896d33f-f407-4db7-9473-75db5411ffb5" containerName="cleanup"
Sep 17 14:30:49.117239 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:49.117220    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-911a82/ipp-route-911a82-kserve-78fb7c6647-zccjm"
Sep 17 14:30:49.120113 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:49.120091    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-911a82\"/\"ipp-route-911a82-kserve-self-signed-certs\""
Sep 17 14:30:49.120113 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:49.120106    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-911a82\"/\"default-dockercfg-m8mpn\""
Sep 17 14:30:49.120282 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:49.120127    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-911a82\"/\"openshift-service-ca.crt\""
Sep 17 14:30:49.120282 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:49.120097    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-911a82\"/\"kube-root-ca.crt\""
Sep 17 14:30:49.127973 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:49.127949    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-911a82/ipp-route-911a82-kserve-78fb7c6647-zccjm"]
Sep 17 14:30:49.165828 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:49.165803    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:30:49.202195 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:49.202167    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3fdb9fc-8cca-4e97-b492-3bf790c235fc-dshm\") pod \"ipp-route-911a82-kserve-78fb7c6647-zccjm\" (UID: \"e3fdb9fc-8cca-4e97-b492-3bf790c235fc\") " pod="ai-tenant-e2e-ipp-a-911a82/ipp-route-911a82-kserve-78fb7c6647-zccjm"
Sep 17 14:30:49.202301 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:49.202195    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3fdb9fc-8cca-4e97-b492-3bf790c235fc-tls-certs\") pod \"ipp-route-911a82-kserve-78fb7c6647-zccjm\" (UID: \"e3fdb9fc-8cca-4e97-b492-3bf790c235fc\") " pod="ai-tenant-e2e-ipp-a-911a82/ipp-route-911a82-kserve-78fb7c6647-zccjm"
Sep 17 14:30:49.202301 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:49.202220    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9nmp\" (UniqueName: \"kubernetes.io/projected/e3fdb9fc-8cca-4e97-b492-3bf790c235fc-kube-api-access-j9nmp\") pod \"ipp-route-911a82-kserve-78fb7c6647-zccjm\" (UID: \"e3fdb9fc-8cca-4e97-b492-3bf790c235fc\") " pod="ai-tenant-e2e-ipp-a-911a82/ipp-route-911a82-kserve-78fb7c6647-zccjm"
Sep 17 14:30:49.202301 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:49.202265    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3fdb9fc-8cca-4e97-b492-3bf790c235fc-home\") pod \"ipp-route-911a82-kserve-78fb7c6647-zccjm\" (UID: \"e3fdb9fc-8cca-4e97-b492-3bf790c235fc\") " pod="ai-tenant-e2e-ipp-a-911a82/ipp-route-911a82-kserve-78fb7c6647-zccjm"
Sep 17 14:30:49.202409 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:49.202352    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3fdb9fc-8cca-4e97-b492-3bf790c235fc-model-cache\") pod \"ipp-route-911a82-kserve-78fb7c6647-zccjm\" (UID: \"e3fdb9fc-8cca-4e97-b492-3bf790c235fc\") " pod="ai-tenant-e2e-ipp-a-911a82/ipp-route-911a82-kserve-78fb7c6647-zccjm"
Sep 17 14:30:49.202409 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:49.202391    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3fdb9fc-8cca-4e97-b492-3bf790c235fc-tmp-dir\") pod \"ipp-route-911a82-kserve-78fb7c6647-zccjm\" (UID: \"e3fdb9fc-8cca-4e97-b492-3bf790c235fc\") " pod="ai-tenant-e2e-ipp-a-911a82/ipp-route-911a82-kserve-78fb7c6647-zccjm"
Sep 17 14:30:49.303135 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:49.303095    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3fdb9fc-8cca-4e97-b492-3bf790c235fc-dshm\") pod \"ipp-route-911a82-kserve-78fb7c6647-zccjm\" (UID: \"e3fdb9fc-8cca-4e97-b492-3bf790c235fc\") " pod="ai-tenant-e2e-ipp-a-911a82/ipp-route-911a82-kserve-78fb7c6647-zccjm"
Sep 17 14:30:49.303135 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:49.303136    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3fdb9fc-8cca-4e97-b492-3bf790c235fc-tls-certs\") pod \"ipp-route-911a82-kserve-78fb7c6647-zccjm\" (UID: \"e3fdb9fc-8cca-4e97-b492-3bf790c235fc\") " pod="ai-tenant-e2e-ipp-a-911a82/ipp-route-911a82-kserve-78fb7c6647-zccjm"
Sep 17 14:30:49.303407 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:49.303163    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9nmp\" (UniqueName: \"kubernetes.io/projected/e3fdb9fc-8cca-4e97-b492-3bf790c235fc-kube-api-access-j9nmp\") pod \"ipp-route-911a82-kserve-78fb7c6647-zccjm\" (UID: \"e3fdb9fc-8cca-4e97-b492-3bf790c235fc\") " pod="ai-tenant-e2e-ipp-a-911a82/ipp-route-911a82-kserve-78fb7c6647-zccjm"
Sep 17 14:30:49.303407 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:49.303230    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3fdb9fc-8cca-4e97-b492-3bf790c235fc-home\") pod \"ipp-route-911a82-kserve-78fb7c6647-zccjm\" (UID: \"e3fdb9fc-8cca-4e97-b492-3bf790c235fc\") " pod="ai-tenant-e2e-ipp-a-911a82/ipp-route-911a82-kserve-78fb7c6647-zccjm"
Sep 17 14:30:49.303407 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:49.303343    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3fdb9fc-8cca-4e97-b492-3bf790c235fc-model-cache\") pod \"ipp-route-911a82-kserve-78fb7c6647-zccjm\" (UID: \"e3fdb9fc-8cca-4e97-b492-3bf790c235fc\") " pod="ai-tenant-e2e-ipp-a-911a82/ipp-route-911a82-kserve-78fb7c6647-zccjm"
Sep 17 14:30:49.303407 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:49.303385    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3fdb9fc-8cca-4e97-b492-3bf790c235fc-tmp-dir\") pod \"ipp-route-911a82-kserve-78fb7c6647-zccjm\" (UID: \"e3fdb9fc-8cca-4e97-b492-3bf790c235fc\") " pod="ai-tenant-e2e-ipp-a-911a82/ipp-route-911a82-kserve-78fb7c6647-zccjm"
Sep 17 14:30:49.303671 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:49.303634    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3fdb9fc-8cca-4e97-b492-3bf790c235fc-home\") pod \"ipp-route-911a82-kserve-78fb7c6647-zccjm\" (UID: \"e3fdb9fc-8cca-4e97-b492-3bf790c235fc\") " pod="ai-tenant-e2e-ipp-a-911a82/ipp-route-911a82-kserve-78fb7c6647-zccjm"
Sep 17 14:30:49.303724 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:49.303684    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3fdb9fc-8cca-4e97-b492-3bf790c235fc-model-cache\") pod \"ipp-route-911a82-kserve-78fb7c6647-zccjm\" (UID: \"e3fdb9fc-8cca-4e97-b492-3bf790c235fc\") " pod="ai-tenant-e2e-ipp-a-911a82/ipp-route-911a82-kserve-78fb7c6647-zccjm"
Sep 17 14:30:49.303784 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:49.303764    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3fdb9fc-8cca-4e97-b492-3bf790c235fc-tmp-dir\") pod \"ipp-route-911a82-kserve-78fb7c6647-zccjm\" (UID: \"e3fdb9fc-8cca-4e97-b492-3bf790c235fc\") " pod="ai-tenant-e2e-ipp-a-911a82/ipp-route-911a82-kserve-78fb7c6647-zccjm"
Sep 17 14:30:49.305567 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:49.305541    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3fdb9fc-8cca-4e97-b492-3bf790c235fc-dshm\") pod \"ipp-route-911a82-kserve-78fb7c6647-zccjm\" (UID: \"e3fdb9fc-8cca-4e97-b492-3bf790c235fc\") " pod="ai-tenant-e2e-ipp-a-911a82/ipp-route-911a82-kserve-78fb7c6647-zccjm"
Sep 17 14:30:49.306218 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:49.306198    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3fdb9fc-8cca-4e97-b492-3bf790c235fc-tls-certs\") pod \"ipp-route-911a82-kserve-78fb7c6647-zccjm\" (UID: \"e3fdb9fc-8cca-4e97-b492-3bf790c235fc\") " pod="ai-tenant-e2e-ipp-a-911a82/ipp-route-911a82-kserve-78fb7c6647-zccjm"
Sep 17 14:30:49.311583 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:49.311558    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9nmp\" (UniqueName: \"kubernetes.io/projected/e3fdb9fc-8cca-4e97-b492-3bf790c235fc-kube-api-access-j9nmp\") pod \"ipp-route-911a82-kserve-78fb7c6647-zccjm\" (UID: \"e3fdb9fc-8cca-4e97-b492-3bf790c235fc\") " pod="ai-tenant-e2e-ipp-a-911a82/ipp-route-911a82-kserve-78fb7c6647-zccjm"
Sep 17 14:30:49.427794 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:49.427724    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-911a82/ipp-route-911a82-kserve-78fb7c6647-zccjm"
Sep 17 14:30:49.705617 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:49.705528    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-911a82/ipp-route-911a82-kserve-78fb7c6647-zccjm"]
Sep 17 14:30:49.710708 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:30:49.710684    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode3fdb9fc_8cca_4e97_b492_3bf790c235fc.slice/crio-7d2e6db14d31a2fdb1a522b3668c037a45431c2dd777852e769029a21a2bec25 WatchSource:0}: Error finding container 7d2e6db14d31a2fdb1a522b3668c037a45431c2dd777852e769029a21a2bec25: Status 404 returned error can't find the container with id 7d2e6db14d31a2fdb1a522b3668c037a45431c2dd777852e769029a21a2bec25
Sep 17 14:30:50.404836 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:50.404796    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-911a82/ipp-route-911a82-kserve-78fb7c6647-zccjm" event={"ID":"e3fdb9fc-8cca-4e97-b492-3bf790c235fc","Type":"ContainerStarted","Data":"be1eb837c9d3c285ea33d8af1db313ccc2a38848d51c1efc9b2586cfa0c1a5fd"}
Sep 17 14:30:50.405226 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:50.404849    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-911a82/ipp-route-911a82-kserve-78fb7c6647-zccjm" event={"ID":"e3fdb9fc-8cca-4e97-b492-3bf790c235fc","Type":"ContainerStarted","Data":"7d2e6db14d31a2fdb1a522b3668c037a45431c2dd777852e769029a21a2bec25"}
Sep 17 14:30:50.424599 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:50.424551    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-911a82/ipp-route-911a82-kserve-78fb7c6647-zccjm" podStartSLOduration=1.424535455 podStartE2EDuration="1.424535455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:30:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:30:50.421334996 +0000 UTC m=+2119.363478638" watchObservedRunningTime="2026-09-17 14:30:50.424535455 +0000 UTC m=+2119.366679100"
Sep 17 14:30:59.428055 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:59.428025    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-911a82/ipp-route-911a82-kserve-78fb7c6647-zccjm"
Sep 17 14:30:59.428546 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:59.428071    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-911a82/ipp-route-911a82-kserve-78fb7c6647-zccjm"
Sep 17 14:30:59.440587 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:59.440564    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-911a82/ipp-route-911a82-kserve-78fb7c6647-zccjm"
Sep 17 14:30:59.478985 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:30:59.478959    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-911a82/ipp-route-911a82-kserve-78fb7c6647-zccjm"
Sep 17 14:31:14.032014 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:14.031970    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:31:14.329810 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:14.329776    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl"]
Sep 17 14:31:14.333488 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:14.333453    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl"
Sep 17 14:31:14.336622 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:14.336600    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-cd2006-openshift-default-dockercfg-9vbrh\""
Sep 17 14:31:14.349145 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:14.349119    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl"]
Sep 17 14:31:14.363285 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:14.363257    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:31:14.442492 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:14.442432    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0-istio-data\") pod \"e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl\" (UID: \"2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0\") " pod="openshift-ingress/e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl"
Sep 17 14:31:14.442700 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:14.442547    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwjkd\" (UniqueName: \"kubernetes.io/projected/2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0-kube-api-access-kwjkd\") pod \"e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl\" (UID: \"2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0\") " pod="openshift-ingress/e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl"
Sep 17 14:31:14.442700 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:14.442587    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0-workload-socket\") pod \"e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl\" (UID: \"2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0\") " pod="openshift-ingress/e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl"
Sep 17 14:31:14.442700 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:14.442617    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0-istio-token\") pod \"e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl\" (UID: \"2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0\") " pod="openshift-ingress/e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl"
Sep 17 14:31:14.442700 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:14.442690    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0-credential-socket\") pod \"e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl\" (UID: \"2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0\") " pod="openshift-ingress/e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl"
Sep 17 14:31:14.442935 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:14.442718    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0-workload-certs\") pod \"e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl\" (UID: \"2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0\") " pod="openshift-ingress/e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl"
Sep 17 14:31:14.442935 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:14.442759    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0-istio-podinfo\") pod \"e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl\" (UID: \"2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0\") " pod="openshift-ingress/e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl"
Sep 17 14:31:14.442935 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:14.442792    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0-istio-envoy\") pod \"e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl\" (UID: \"2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0\") " pod="openshift-ingress/e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl"
Sep 17 14:31:14.442935 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:14.442826    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0-istiod-ca-cert\") pod \"e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl\" (UID: \"2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0\") " pod="openshift-ingress/e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl"
Sep 17 14:31:14.539102 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:14.539058    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:31:14.544203 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:14.544182    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0-istio-data\") pod \"e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl\" (UID: \"2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0\") " pod="openshift-ingress/e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl"
Sep 17 14:31:14.544280 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:14.544229    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwjkd\" (UniqueName: \"kubernetes.io/projected/2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0-kube-api-access-kwjkd\") pod \"e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl\" (UID: \"2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0\") " pod="openshift-ingress/e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl"
Sep 17 14:31:14.544383 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:14.544363    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0-workload-socket\") pod \"e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl\" (UID: \"2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0\") " pod="openshift-ingress/e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl"
Sep 17 14:31:14.544423 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:14.544406    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0-istio-token\") pod \"e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl\" (UID: \"2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0\") " pod="openshift-ingress/e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl"
Sep 17 14:31:14.544498 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:14.544453    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0-credential-socket\") pod \"e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl\" (UID: \"2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0\") " pod="openshift-ingress/e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl"
Sep 17 14:31:14.544498 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:14.544489    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0-workload-certs\") pod \"e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl\" (UID: \"2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0\") " pod="openshift-ingress/e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl"
Sep 17 14:31:14.544611 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:14.544518    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0-istio-podinfo\") pod \"e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl\" (UID: \"2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0\") " pod="openshift-ingress/e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl"
Sep 17 14:31:14.544666 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:14.544610    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0-istio-data\") pod \"e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl\" (UID: \"2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0\") " pod="openshift-ingress/e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl"
Sep 17 14:31:14.544666 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:14.544634    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0-istio-envoy\") pod \"e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl\" (UID: \"2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0\") " pod="openshift-ingress/e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl"
Sep 17 14:31:14.544774 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:14.544707    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0-istiod-ca-cert\") pod \"e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl\" (UID: \"2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0\") " pod="openshift-ingress/e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl"
Sep 17 14:31:14.544830 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:14.544798    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0-workload-socket\") pod \"e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl\" (UID: \"2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0\") " pod="openshift-ingress/e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl"
Sep 17 14:31:14.544883 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:14.544848    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0-credential-socket\") pod \"e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl\" (UID: \"2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0\") " pod="openshift-ingress/e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl"
Sep 17 14:31:14.544959 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:14.544924    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0-workload-certs\") pod \"e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl\" (UID: \"2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0\") " pod="openshift-ingress/e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl"
Sep 17 14:31:14.545365 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:14.545346    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0-istiod-ca-cert\") pod \"e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl\" (UID: \"2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0\") " pod="openshift-ingress/e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl"
Sep 17 14:31:14.546832 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:14.546811    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0-istio-envoy\") pod \"e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl\" (UID: \"2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0\") " pod="openshift-ingress/e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl"
Sep 17 14:31:14.546941 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:14.546925    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0-istio-podinfo\") pod \"e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl\" (UID: \"2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0\") " pod="openshift-ingress/e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl"
Sep 17 14:31:14.557511 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:14.557455    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0-istio-token\") pod \"e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl\" (UID: \"2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0\") " pod="openshift-ingress/e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl"
Sep 17 14:31:14.560372 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:14.560349    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwjkd\" (UniqueName: \"kubernetes.io/projected/2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0-kube-api-access-kwjkd\") pod \"e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl\" (UID: \"2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0\") " pod="openshift-ingress/e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl"
Sep 17 14:31:14.644277 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:14.644198    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl"
Sep 17 14:31:14.926966 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:14.926941    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl"]
Sep 17 14:31:14.929285 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:31:14.929259    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2405ae78_bb3d_45c7_8bdb_a29fe98cd8d0.slice/crio-9e0012de10f4808c852e8e2c7d35158e2faed714135edb3b1bf4b94d6f020b51 WatchSource:0}: Error finding container 9e0012de10f4808c852e8e2c7d35158e2faed714135edb3b1bf4b94d6f020b51: Status 404 returned error can't find the container with id 9e0012de10f4808c852e8e2c7d35158e2faed714135edb3b1bf4b94d6f020b51
Sep 17 14:31:14.931583 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:14.931554    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:31:14.931644 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:14.931615    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:31:14.931686 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:14.931641    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:31:15.570919 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:15.570887    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl" event={"ID":"2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0","Type":"ContainerStarted","Data":"9efe2becdd65b0814f4528d1348da27934db594947ef799adfd87ee07dccb835"}
Sep 17 14:31:15.571279 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:15.570935    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl" event={"ID":"2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0","Type":"ContainerStarted","Data":"9e0012de10f4808c852e8e2c7d35158e2faed714135edb3b1bf4b94d6f020b51"}
Sep 17 14:31:15.594473 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:15.594412    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl" podStartSLOduration=1.5943967940000001 podStartE2EDuration="1.594396794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:31:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:31:15.592456564 +0000 UTC m=+2144.534600214" watchObservedRunningTime="2026-09-17 14:31:15.594396794 +0000 UTC m=+2144.536540439"
Sep 17 14:31:15.645259 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:15.645234    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl"
Sep 17 14:31:15.650075 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:15.650052    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl"
Sep 17 14:31:16.577832 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:16.577799    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl"
Sep 17 14:31:16.578739 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:16.578724    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl"
Sep 17 14:31:17.466958 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:17.466900    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:31:17.580449 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:17.580401    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:31:17.677736 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:17.677687    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:31:18.770931 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:18.770898    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:31:19.612905 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:19.612841    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:31:20.550797 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:20.550758    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:31:21.437471 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:21.437409    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:31:21.553496 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:21.553442    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:31:21.655363 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:21.655328    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:31:21.955790 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:21.955762    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:31:22.057673 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:22.057625    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:31:32.155204 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:32.155121    2560 scope.go:117] "RemoveContainer" containerID="d5c055ba267f752b8193cb6a7b8302ab62325235ede07f42f9b3b9f524b424d1"
Sep 17 14:31:32.174672 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:32.174640    2560 scope.go:117] "RemoveContainer" containerID="960f6baef91ecb59711855aee9510d37eaacde69d6f60d379dd7bf00bf87d9d5"
Sep 17 14:31:32.187578 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:32.187558    2560 scope.go:117] "RemoveContainer" containerID="849cadc71389223ce66a35f570aa8a1e1179652601076f0159c043596fa8276b"
Sep 17 14:31:32.195991 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:32.195969    2560 scope.go:117] "RemoveContainer" containerID="50ea7864495aa34217865296bc4db68b5473ed5b503c5c346021f98ffc032e35"
Sep 17 14:31:36.729322 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:36.729293    2560 generic.go:356] "Generic (PLEG): container finished" podID="2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0" containerID="9efe2becdd65b0814f4528d1348da27934db594947ef799adfd87ee07dccb835" exitCode=0
Sep 17 14:31:36.729702 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:36.729369    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl" event={"ID":"2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0","Type":"ContainerDied","Data":"9efe2becdd65b0814f4528d1348da27934db594947ef799adfd87ee07dccb835"}
Sep 17 14:31:36.729814 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:36.729800    2560 scope.go:117] "RemoveContainer" containerID="9efe2becdd65b0814f4528d1348da27934db594947ef799adfd87ee07dccb835"
Sep 17 14:31:36.730928 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:36.730898    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:31:36.731025 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:36.730980    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:31:36.731087 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:36.731028    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:31:37.737477 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:37.737428    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl" event={"ID":"2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0","Type":"ContainerStarted","Data":"93457e5607ae05212b67bb6ef8cf40d01f3d175440857f5d54201e74652a8175"}
Sep 17 14:31:38.644687 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:38.644654    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl"
Sep 17 14:31:38.644887 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:38.644839    2560 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Sep 17 14:31:38.644969 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:38.644911    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl" podUID="2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Sep 17 14:31:39.645389 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:39.645354    2560 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Sep 17 14:31:39.645789 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:39.645433    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl" podUID="2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Sep 17 14:31:40.644706 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:40.644668    2560 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Sep 17 14:31:40.644886 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:40.644723    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl" podUID="2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Sep 17 14:31:41.645016 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:41.644982    2560 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Sep 17 14:31:41.645401 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:41.645033    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl" podUID="2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Sep 17 14:31:42.644862 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:42.644826    2560 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Sep 17 14:31:42.645041 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:42.644889    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl" podUID="2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Sep 17 14:31:43.649096 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:43.649063    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl"
Sep 17 14:31:43.649491 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:43.649337    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl"
Sep 17 14:31:43.650064 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:43.650048    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl"
Sep 17 14:31:45.282752 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:45.282721    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:31:46.524702 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:46.524660    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:31:47.056548 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:47.056506    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:31:47.786718 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:47.786685    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:31:47.912399 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:47.912370    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:31:48.534982 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:48.534946    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-cd2006-nmxlb"]
Sep 17 14:31:48.538874 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:48.538856    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-cd2006-nmxlb"
Sep 17 14:31:48.541591 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:48.541567    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rp9ww\""
Sep 17 14:31:48.542990 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:48.542969    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 14:31:48.556776 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:48.556746    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-cd2006-nmxlb"]
Sep 17 14:31:48.602224 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:48.602188    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c71a28fc-c2c3-4bf9-adc0-84337518e849-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-cd2006-nmxlb\" (UID: \"c71a28fc-c2c3-4bf9-adc0-84337518e849\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-cd2006-nmxlb"
Sep 17 14:31:48.703454 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:48.703417    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c71a28fc-c2c3-4bf9-adc0-84337518e849-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-cd2006-nmxlb\" (UID: \"c71a28fc-c2c3-4bf9-adc0-84337518e849\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-cd2006-nmxlb"
Sep 17 14:31:48.704045 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:48.704023    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c71a28fc-c2c3-4bf9-adc0-84337518e849-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-cd2006-nmxlb\" (UID: \"c71a28fc-c2c3-4bf9-adc0-84337518e849\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-cd2006-nmxlb"
Sep 17 14:31:48.849675 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:48.849644    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-cd2006-nmxlb"
Sep 17 14:31:48.998323 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:48.998284    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:31:49.125086 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:49.125054    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-cd2006-nmxlb"]
Sep 17 14:31:49.126020 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:31:49.125992    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc71a28fc_c2c3_4bf9_adc0_84337518e849.slice/crio-c4a6662564bd50db5f0ae9faee96d5e8c67fcd498ff7373f50948a4bdad78953 WatchSource:0}: Error finding container c4a6662564bd50db5f0ae9faee96d5e8c67fcd498ff7373f50948a4bdad78953: Status 404 returned error can't find the container with id c4a6662564bd50db5f0ae9faee96d5e8c67fcd498ff7373f50948a4bdad78953
Sep 17 14:31:49.758669 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:49.758588    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:31:49.819649 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:49.818986    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-cd2006-nmxlb" event={"ID":"c71a28fc-c2c3-4bf9-adc0-84337518e849","Type":"ContainerStarted","Data":"087303a4071a9a6f630ba6adeadfd6c886a650fdc64d7a982e61ecb3ac6f31d0"}
Sep 17 14:31:49.819649 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:49.819234    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-cd2006-nmxlb" event={"ID":"c71a28fc-c2c3-4bf9-adc0-84337518e849","Type":"ContainerStarted","Data":"c4a6662564bd50db5f0ae9faee96d5e8c67fcd498ff7373f50948a4bdad78953"}
Sep 17 14:31:49.844636 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:49.844590    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-cd2006-nmxlb" podStartSLOduration=1.844576518 podStartE2EDuration="1.844576518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:31:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:31:49.844053525 +0000 UTC m=+2178.786197170" watchObservedRunningTime="2026-09-17 14:31:49.844576518 +0000 UTC m=+2178.786720162"
Sep 17 14:31:50.636971 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:50.636930    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:31:50.826448 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:50.826417    2560 generic.go:356] "Generic (PLEG): container finished" podID="c71a28fc-c2c3-4bf9-adc0-84337518e849" containerID="087303a4071a9a6f630ba6adeadfd6c886a650fdc64d7a982e61ecb3ac6f31d0" exitCode=0
Sep 17 14:31:50.826654 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:50.826495    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-cd2006-nmxlb" event={"ID":"c71a28fc-c2c3-4bf9-adc0-84337518e849","Type":"ContainerDied","Data":"087303a4071a9a6f630ba6adeadfd6c886a650fdc64d7a982e61ecb3ac6f31d0"}
Sep 17 14:31:51.444875 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:51.444843    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:31:51.964747 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:51.964719    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-cd2006-nmxlb"
Sep 17 14:31:52.038508 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:52.038479    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c71a28fc-c2c3-4bf9-adc0-84337518e849-maas-api-service-ca\") pod \"c71a28fc-c2c3-4bf9-adc0-84337518e849\" (UID: \"c71a28fc-c2c3-4bf9-adc0-84337518e849\") "
Sep 17 14:31:52.038760 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:52.038738    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c71a28fc-c2c3-4bf9-adc0-84337518e849-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c71a28fc-c2c3-4bf9-adc0-84337518e849" (UID: "c71a28fc-c2c3-4bf9-adc0-84337518e849"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:31:52.038832 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:52.038820    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c71a28fc-c2c3-4bf9-adc0-84337518e849-maas-api-service-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:31:52.460187 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:52.460144    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:31:52.839732 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:52.839699    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-cd2006-nmxlb"
Sep 17 14:31:52.839908 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:52.839732    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-cd2006-nmxlb" event={"ID":"c71a28fc-c2c3-4bf9-adc0-84337518e849","Type":"ContainerDied","Data":"c4a6662564bd50db5f0ae9faee96d5e8c67fcd498ff7373f50948a4bdad78953"}
Sep 17 14:31:52.839908 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:52.839782    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4a6662564bd50db5f0ae9faee96d5e8c67fcd498ff7373f50948a4bdad78953"
Sep 17 14:31:53.868251 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:53.868202    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:31:59.148665 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:59.148611    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:31:59.827050 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:59.827016    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:31:59.908065 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:59.908033    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:31:59.975454 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:31:59.975415    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:32:00.056148 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:00.056104    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-cd2006-nmxlb"]
Sep 17 14:32:00.056369 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:00.056173    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-cd2006-nmxlb"]
Sep 17 14:32:00.058002 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:00.057973    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:32:00.135133 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:00.135042    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:32:00.453336 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:00.453244    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:32:00.553827 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:00.553792    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:32:01.348906 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:01.348866    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:32:01.570190 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:01.570159    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c71a28fc-c2c3-4bf9-adc0-84337518e849" path="/var/lib/kubelet/pods/c71a28fc-c2c3-4bf9-adc0-84337518e849/volumes"
Sep 17 14:32:02.461777 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:02.461738    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:32:03.381536 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:03.381501    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:32:04.227038 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:04.227004    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:32:06.906611 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:06.906574    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:32:07.183968 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:07.183879    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl"]
Sep 17 14:32:07.184144 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:07.184122    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl" podUID="2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0" containerName="istio-proxy" containerID="cri-o://93457e5607ae05212b67bb6ef8cf40d01f3d175440857f5d54201e74652a8175" gracePeriod=30
Sep 17 14:32:07.461772 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:07.461676    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:32:07.524903 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:07.524867    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:32:11.119399 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:11.119360    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-911a82-zkk2z"]
Sep 17 14:32:11.119831 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:11.119816    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c71a28fc-c2c3-4bf9-adc0-84337518e849" containerName="revoke-keys"
Sep 17 14:32:11.119876 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:11.119832    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c71a28fc-c2c3-4bf9-adc0-84337518e849" containerName="revoke-keys"
Sep 17 14:32:11.119957 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:11.119907    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="c71a28fc-c2c3-4bf9-adc0-84337518e849" containerName="revoke-keys"
Sep 17 14:32:11.123037 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:11.123022    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-911a82-zkk2z"
Sep 17 14:32:11.126171 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:11.126149    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 14:32:11.126894 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:11.126876    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rp9ww\""
Sep 17 14:32:11.132616 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:11.132595    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-911a82-zkk2z"]
Sep 17 14:32:11.142841 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:11.142809    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c03532a3-a9fb-4c4e-af43-8df2c342cad6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-911a82-zkk2z\" (UID: \"c03532a3-a9fb-4c4e-af43-8df2c342cad6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-911a82-zkk2z"
Sep 17 14:32:11.244026 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:11.243989    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c03532a3-a9fb-4c4e-af43-8df2c342cad6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-911a82-zkk2z\" (UID: \"c03532a3-a9fb-4c4e-af43-8df2c342cad6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-911a82-zkk2z"
Sep 17 14:32:11.244720 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:11.244698    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c03532a3-a9fb-4c4e-af43-8df2c342cad6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-911a82-zkk2z\" (UID: \"c03532a3-a9fb-4c4e-af43-8df2c342cad6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-911a82-zkk2z"
Sep 17 14:32:11.434760 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:11.434663    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-911a82-zkk2z"
Sep 17 14:32:11.459670 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:11.459627    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:32:11.711230 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:11.711198    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-911a82-zkk2z"]
Sep 17 14:32:11.712729 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:32:11.712702    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc03532a3_a9fb_4c4e_af43_8df2c342cad6.slice/crio-a1c22a29d3cb2bf2822d0f6b58c24ee7adf90036050aef69af87a27a2635140f WatchSource:0}: Error finding container a1c22a29d3cb2bf2822d0f6b58c24ee7adf90036050aef69af87a27a2635140f: Status 404 returned error can't find the container with id a1c22a29d3cb2bf2822d0f6b58c24ee7adf90036050aef69af87a27a2635140f
Sep 17 14:32:11.961682 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:11.961598    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-911a82-zkk2z" event={"ID":"c03532a3-a9fb-4c4e-af43-8df2c342cad6","Type":"ContainerStarted","Data":"a1c22a29d3cb2bf2822d0f6b58c24ee7adf90036050aef69af87a27a2635140f"}
Sep 17 14:32:12.368639 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:12.368568    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:32:12.425993 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:12.425968    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl"
Sep 17 14:32:12.456563 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:12.456526    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0-workload-certs\") pod \"2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0\" (UID: \"2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0\") "
Sep 17 14:32:12.456724 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:12.456575    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0-credential-socket\") pod \"2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0\" (UID: \"2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0\") "
Sep 17 14:32:12.456724 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:12.456645    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0-istio-podinfo\") pod \"2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0\" (UID: \"2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0\") "
Sep 17 14:32:12.456724 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:12.456672    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0-istio-token\") pod \"2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0\" (UID: \"2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0\") "
Sep 17 14:32:12.456885 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:12.456725    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0-istio-data\") pod \"2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0\" (UID: \"2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0\") "
Sep 17 14:32:12.456885 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:12.456829    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0-istiod-ca-cert\") pod \"2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0\" (UID: \"2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0\") "
Sep 17 14:32:12.456885 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:12.456844    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0" (UID: "2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:32:12.456885 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:12.456856    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0" (UID: "2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:32:12.456885 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:12.456874    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0-workload-socket\") pod \"2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0\" (UID: \"2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0\") "
Sep 17 14:32:12.457162 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:12.457002    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kwjkd\" (UniqueName: \"kubernetes.io/projected/2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0-kube-api-access-kwjkd\") pod \"2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0\" (UID: \"2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0\") "
Sep 17 14:32:12.457162 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:12.457037    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0-istio-envoy\") pod \"2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0\" (UID: \"2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0\") "
Sep 17 14:32:12.457162 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:12.457112    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0" (UID: "2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:32:12.457338 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:12.457168    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0" (UID: "2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:32:12.457401 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:12.457352    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0-istio-data" (OuterVolumeSpecName: "istio-data") pod "2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0" (UID: "2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:32:12.457530 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:12.457510    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0-workload-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:32:12.457606 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:12.457534    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0-credential-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:32:12.457606 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:12.457547    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0-istio-data\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:32:12.457606 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:12.457559    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0-istiod-ca-cert\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:32:12.457606 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:12.457570    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0-workload-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:32:12.459688 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:12.459651    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0-istio-token" (OuterVolumeSpecName: "istio-token") pod "2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0" (UID: "2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:32:12.460001 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:12.459968    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0" (UID: "2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 14:32:12.461006 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:12.460875    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0" (UID: "2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:32:12.461205 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:12.461179    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0-kube-api-access-kwjkd" (OuterVolumeSpecName: "kube-api-access-kwjkd") pod "2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0" (UID: "2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0"). InnerVolumeSpecName "kube-api-access-kwjkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:32:12.558491 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:12.558367    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0-istio-podinfo\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:32:12.558491 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:12.558400    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0-istio-token\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:32:12.558491 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:12.558410    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwjkd\" (UniqueName: \"kubernetes.io/projected/2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0-kube-api-access-kwjkd\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:32:12.558491 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:12.558420    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0-istio-envoy\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:32:12.969087 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:12.968991    2560 generic.go:356] "Generic (PLEG): container finished" podID="c03532a3-a9fb-4c4e-af43-8df2c342cad6" containerID="9861848a84c6d7a4a774fb46e165129821225efde7ad9e9bf543f200222abdda" exitCode=0
Sep 17 14:32:12.969266 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:12.969084    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-911a82-zkk2z" event={"ID":"c03532a3-a9fb-4c4e-af43-8df2c342cad6","Type":"ContainerDied","Data":"9861848a84c6d7a4a774fb46e165129821225efde7ad9e9bf543f200222abdda"}
Sep 17 14:32:12.970798 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:12.970774    2560 generic.go:356] "Generic (PLEG): container finished" podID="2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0" containerID="93457e5607ae05212b67bb6ef8cf40d01f3d175440857f5d54201e74652a8175" exitCode=0
Sep 17 14:32:12.970929 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:12.970820    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl"
Sep 17 14:32:12.970929 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:12.970853    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl" event={"ID":"2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0","Type":"ContainerDied","Data":"93457e5607ae05212b67bb6ef8cf40d01f3d175440857f5d54201e74652a8175"}
Sep 17 14:32:12.970929 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:12.970902    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl" event={"ID":"2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0","Type":"ContainerDied","Data":"9e0012de10f4808c852e8e2c7d35158e2faed714135edb3b1bf4b94d6f020b51"}
Sep 17 14:32:12.971056 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:12.970937    2560 scope.go:117] "RemoveContainer" containerID="93457e5607ae05212b67bb6ef8cf40d01f3d175440857f5d54201e74652a8175"
Sep 17 14:32:12.986175 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:12.986155    2560 scope.go:117] "RemoveContainer" containerID="9efe2becdd65b0814f4528d1348da27934db594947ef799adfd87ee07dccb835"
Sep 17 14:32:12.996026 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:12.996007    2560 scope.go:117] "RemoveContainer" containerID="93457e5607ae05212b67bb6ef8cf40d01f3d175440857f5d54201e74652a8175"
Sep 17 14:32:12.996363 ip-10-0-129-158 kubenswrapper[2560]: E0917 14:32:12.996342    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93457e5607ae05212b67bb6ef8cf40d01f3d175440857f5d54201e74652a8175\": container with ID starting with 93457e5607ae05212b67bb6ef8cf40d01f3d175440857f5d54201e74652a8175 not found: ID does not exist" containerID="93457e5607ae05212b67bb6ef8cf40d01f3d175440857f5d54201e74652a8175"
Sep 17 14:32:12.996411 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:12.996372    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93457e5607ae05212b67bb6ef8cf40d01f3d175440857f5d54201e74652a8175"} err="failed to get container status \"93457e5607ae05212b67bb6ef8cf40d01f3d175440857f5d54201e74652a8175\": rpc error: code = NotFound desc = could not find container \"93457e5607ae05212b67bb6ef8cf40d01f3d175440857f5d54201e74652a8175\": container with ID starting with 93457e5607ae05212b67bb6ef8cf40d01f3d175440857f5d54201e74652a8175 not found: ID does not exist"
Sep 17 14:32:12.996411 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:12.996390    2560 scope.go:117] "RemoveContainer" containerID="9efe2becdd65b0814f4528d1348da27934db594947ef799adfd87ee07dccb835"
Sep 17 14:32:12.996710 ip-10-0-129-158 kubenswrapper[2560]: E0917 14:32:12.996692    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9efe2becdd65b0814f4528d1348da27934db594947ef799adfd87ee07dccb835\": container with ID starting with 9efe2becdd65b0814f4528d1348da27934db594947ef799adfd87ee07dccb835 not found: ID does not exist" containerID="9efe2becdd65b0814f4528d1348da27934db594947ef799adfd87ee07dccb835"
Sep 17 14:32:12.996766 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:12.996715    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9efe2becdd65b0814f4528d1348da27934db594947ef799adfd87ee07dccb835"} err="failed to get container status \"9efe2becdd65b0814f4528d1348da27934db594947ef799adfd87ee07dccb835\": rpc error: code = NotFound desc = could not find container \"9efe2becdd65b0814f4528d1348da27934db594947ef799adfd87ee07dccb835\": container with ID starting with 9efe2becdd65b0814f4528d1348da27934db594947ef799adfd87ee07dccb835 not found: ID does not exist"
Sep 17 14:32:13.009008 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:13.008977    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl"]
Sep 17 14:32:13.012653 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:13.012629    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-cd2006-openshift-default-5f89779d7d-nqqfl"]
Sep 17 14:32:13.243652 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:13.243552    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:32:13.570144 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:13.570107    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0" path="/var/lib/kubelet/pods/2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0/volumes"
Sep 17 14:32:14.109826 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:14.109794    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-911a82-zkk2z"
Sep 17 14:32:14.149083 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:14.149041    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:32:14.176283 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:14.176245    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c03532a3-a9fb-4c4e-af43-8df2c342cad6-maas-api-service-ca\") pod \"c03532a3-a9fb-4c4e-af43-8df2c342cad6\" (UID: \"c03532a3-a9fb-4c4e-af43-8df2c342cad6\") "
Sep 17 14:32:14.176643 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:14.176620    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c03532a3-a9fb-4c4e-af43-8df2c342cad6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c03532a3-a9fb-4c4e-af43-8df2c342cad6" (UID: "c03532a3-a9fb-4c4e-af43-8df2c342cad6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:32:14.277561 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:14.277448    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c03532a3-a9fb-4c4e-af43-8df2c342cad6-maas-api-service-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:32:14.928412 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:14.928377    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:32:14.989723 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:14.989693    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-911a82-zkk2z"
Sep 17 14:32:14.989723 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:14.989719    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-911a82-zkk2z" event={"ID":"c03532a3-a9fb-4c4e-af43-8df2c342cad6","Type":"ContainerDied","Data":"a1c22a29d3cb2bf2822d0f6b58c24ee7adf90036050aef69af87a27a2635140f"}
Sep 17 14:32:14.989932 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:14.989761    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1c22a29d3cb2bf2822d0f6b58c24ee7adf90036050aef69af87a27a2635140f"
Sep 17 14:32:15.926141 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:15.926107    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:32:26.975844 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:26.975810    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:32:27.050290 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:27.050257    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:32:27.110659 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:27.110625    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:32:27.214916 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:27.214877    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:32:27.217176 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:27.217149    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-911a82-zkk2z"]
Sep 17 14:32:27.220067 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:27.220043    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-911a82-zkk2z"]
Sep 17 14:32:27.302655 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:27.302611    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:32:27.569354 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:27.569277    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c03532a3-a9fb-4c4e-af43-8df2c342cad6" path="/var/lib/kubelet/pods/c03532a3-a9fb-4c4e-af43-8df2c342cad6/volumes"
Sep 17 14:32:27.641486 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:27.641433    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:32:27.743100 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:27.743061    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:32:28.581946 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:28.581913    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:32:29.827892 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:29.827861    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:32:30.722691 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:30.722658    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:32:32.237815 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:32.237777    2560 scope.go:117] "RemoveContainer" containerID="b0fa5a624a11a8db3db07f630d8211ba0dc40f58ff2f622e979496811c7d95de"
Sep 17 14:32:32.248203 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:32.248169    2560 scope.go:117] "RemoveContainer" containerID="f66bac8bc10a559a4960182bf6d46b44572d1ec71e6049c2a858f0308eae8667"
Sep 17 14:32:32.257798 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:32.257778    2560 scope.go:117] "RemoveContainer" containerID="bd5a80e72752ded4c659858250ac3731838b2e9f07f8e20218cda2d8f898e049"
Sep 17 14:32:32.620481 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:32.620433    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-911a82/ipp-route-911a82-kserve-78fb7c6647-zccjm"]
Sep 17 14:32:32.703527 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:32.703493    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:32:36.100351 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:36.100311    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:32:47.639992 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:47.639954    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-911a82/ipp-route-911a82-kserve-78fb7c6647-zccjm" podUID="e3fdb9fc-8cca-4e97-b492-3bf790c235fc" containerName="main" containerID="cri-o://be1eb837c9d3c285ea33d8af1db313ccc2a38848d51c1efc9b2586cfa0c1a5fd" gracePeriod=45
Sep 17 14:32:47.890109 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:47.890048    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-911a82/ipp-route-911a82-kserve-78fb7c6647-zccjm"
Sep 17 14:32:47.984549 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:47.984522    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3fdb9fc-8cca-4e97-b492-3bf790c235fc-tls-certs\") pod \"e3fdb9fc-8cca-4e97-b492-3bf790c235fc\" (UID: \"e3fdb9fc-8cca-4e97-b492-3bf790c235fc\") "
Sep 17 14:32:47.984709 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:47.984627    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3fdb9fc-8cca-4e97-b492-3bf790c235fc-dshm\") pod \"e3fdb9fc-8cca-4e97-b492-3bf790c235fc\" (UID: \"e3fdb9fc-8cca-4e97-b492-3bf790c235fc\") "
Sep 17 14:32:47.984709 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:47.984659    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3fdb9fc-8cca-4e97-b492-3bf790c235fc-model-cache\") pod \"e3fdb9fc-8cca-4e97-b492-3bf790c235fc\" (UID: \"e3fdb9fc-8cca-4e97-b492-3bf790c235fc\") "
Sep 17 14:32:47.984709 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:47.984677    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3fdb9fc-8cca-4e97-b492-3bf790c235fc-tmp-dir\") pod \"e3fdb9fc-8cca-4e97-b492-3bf790c235fc\" (UID: \"e3fdb9fc-8cca-4e97-b492-3bf790c235fc\") "
Sep 17 14:32:47.984876 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:47.984712    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j9nmp\" (UniqueName: \"kubernetes.io/projected/e3fdb9fc-8cca-4e97-b492-3bf790c235fc-kube-api-access-j9nmp\") pod \"e3fdb9fc-8cca-4e97-b492-3bf790c235fc\" (UID: \"e3fdb9fc-8cca-4e97-b492-3bf790c235fc\") "
Sep 17 14:32:47.984876 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:47.984770    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3fdb9fc-8cca-4e97-b492-3bf790c235fc-home\") pod \"e3fdb9fc-8cca-4e97-b492-3bf790c235fc\" (UID: \"e3fdb9fc-8cca-4e97-b492-3bf790c235fc\") "
Sep 17 14:32:47.984988 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:47.984921    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3fdb9fc-8cca-4e97-b492-3bf790c235fc-model-cache" (OuterVolumeSpecName: "model-cache") pod "e3fdb9fc-8cca-4e97-b492-3bf790c235fc" (UID: "e3fdb9fc-8cca-4e97-b492-3bf790c235fc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:32:47.984988 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:47.984940    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3fdb9fc-8cca-4e97-b492-3bf790c235fc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e3fdb9fc-8cca-4e97-b492-3bf790c235fc" (UID: "e3fdb9fc-8cca-4e97-b492-3bf790c235fc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:32:47.985141 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:47.985112    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3fdb9fc-8cca-4e97-b492-3bf790c235fc-home" (OuterVolumeSpecName: "home") pod "e3fdb9fc-8cca-4e97-b492-3bf790c235fc" (UID: "e3fdb9fc-8cca-4e97-b492-3bf790c235fc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:32:47.985214 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:47.985184    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3fdb9fc-8cca-4e97-b492-3bf790c235fc-model-cache\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:32:47.985269 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:47.985222    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3fdb9fc-8cca-4e97-b492-3bf790c235fc-tmp-dir\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:32:47.986776 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:47.986746    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3fdb9fc-8cca-4e97-b492-3bf790c235fc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e3fdb9fc-8cca-4e97-b492-3bf790c235fc" (UID: "e3fdb9fc-8cca-4e97-b492-3bf790c235fc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:32:47.986776 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:47.986764    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3fdb9fc-8cca-4e97-b492-3bf790c235fc-dshm" (OuterVolumeSpecName: "dshm") pod "e3fdb9fc-8cca-4e97-b492-3bf790c235fc" (UID: "e3fdb9fc-8cca-4e97-b492-3bf790c235fc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:32:47.986776 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:47.986746    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3fdb9fc-8cca-4e97-b492-3bf790c235fc-kube-api-access-j9nmp" (OuterVolumeSpecName: "kube-api-access-j9nmp") pod "e3fdb9fc-8cca-4e97-b492-3bf790c235fc" (UID: "e3fdb9fc-8cca-4e97-b492-3bf790c235fc"). InnerVolumeSpecName "kube-api-access-j9nmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:32:48.085775 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:48.085751    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3fdb9fc-8cca-4e97-b492-3bf790c235fc-dshm\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:32:48.085775 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:48.085772    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9nmp\" (UniqueName: \"kubernetes.io/projected/e3fdb9fc-8cca-4e97-b492-3bf790c235fc-kube-api-access-j9nmp\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:32:48.085928 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:48.085783    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3fdb9fc-8cca-4e97-b492-3bf790c235fc-home\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:32:48.085928 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:48.085792    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3fdb9fc-8cca-4e97-b492-3bf790c235fc-tls-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:32:48.190353 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:48.190295    2560 generic.go:356] "Generic (PLEG): container finished" podID="e3fdb9fc-8cca-4e97-b492-3bf790c235fc" containerID="be1eb837c9d3c285ea33d8af1db313ccc2a38848d51c1efc9b2586cfa0c1a5fd" exitCode=0
Sep 17 14:32:48.190353 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:48.190333    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-911a82/ipp-route-911a82-kserve-78fb7c6647-zccjm"
Sep 17 14:32:48.190534 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:48.190373    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-911a82/ipp-route-911a82-kserve-78fb7c6647-zccjm" event={"ID":"e3fdb9fc-8cca-4e97-b492-3bf790c235fc","Type":"ContainerDied","Data":"be1eb837c9d3c285ea33d8af1db313ccc2a38848d51c1efc9b2586cfa0c1a5fd"}
Sep 17 14:32:48.190534 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:48.190417    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-911a82/ipp-route-911a82-kserve-78fb7c6647-zccjm" event={"ID":"e3fdb9fc-8cca-4e97-b492-3bf790c235fc","Type":"ContainerDied","Data":"7d2e6db14d31a2fdb1a522b3668c037a45431c2dd777852e769029a21a2bec25"}
Sep 17 14:32:48.190534 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:48.190434    2560 scope.go:117] "RemoveContainer" containerID="be1eb837c9d3c285ea33d8af1db313ccc2a38848d51c1efc9b2586cfa0c1a5fd"
Sep 17 14:32:48.200122 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:48.200102    2560 scope.go:117] "RemoveContainer" containerID="be1eb837c9d3c285ea33d8af1db313ccc2a38848d51c1efc9b2586cfa0c1a5fd"
Sep 17 14:32:48.200400 ip-10-0-129-158 kubenswrapper[2560]: E0917 14:32:48.200379    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be1eb837c9d3c285ea33d8af1db313ccc2a38848d51c1efc9b2586cfa0c1a5fd\": container with ID starting with be1eb837c9d3c285ea33d8af1db313ccc2a38848d51c1efc9b2586cfa0c1a5fd not found: ID does not exist" containerID="be1eb837c9d3c285ea33d8af1db313ccc2a38848d51c1efc9b2586cfa0c1a5fd"
Sep 17 14:32:48.200495 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:48.200407    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be1eb837c9d3c285ea33d8af1db313ccc2a38848d51c1efc9b2586cfa0c1a5fd"} err="failed to get container status \"be1eb837c9d3c285ea33d8af1db313ccc2a38848d51c1efc9b2586cfa0c1a5fd\": rpc error: code = NotFound desc = could not find container \"be1eb837c9d3c285ea33d8af1db313ccc2a38848d51c1efc9b2586cfa0c1a5fd\": container with ID starting with be1eb837c9d3c285ea33d8af1db313ccc2a38848d51c1efc9b2586cfa0c1a5fd not found: ID does not exist"
Sep 17 14:32:48.213413 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:48.213388    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-911a82/ipp-route-911a82-kserve-78fb7c6647-zccjm"]
Sep 17 14:32:48.217199 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:48.217180    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-911a82/ipp-route-911a82-kserve-78fb7c6647-zccjm"]
Sep 17 14:32:49.569793 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:49.569757    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3fdb9fc-8cca-4e97-b492-3bf790c235fc" path="/var/lib/kubelet/pods/e3fdb9fc-8cca-4e97-b492-3bf790c235fc/volumes"
Sep 17 14:32:49.672249 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:49.672216    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:32:55.001430 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:55.001335    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:32:56.583619 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:32:56.583579    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:02.803919 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:02.803872    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:03.120932 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:03.120837    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf"]
Sep 17 14:33:03.121105 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:03.121080    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf" podUID="4633a212-1cff-495c-80df-ceaa196237ae" containerName="istio-proxy" containerID="cri-o://9a9723be0ff9a613fabb68ecebcd627f3e5e896183b0d5e02bc4a601b982ab43" gracePeriod=30
Sep 17 14:33:03.572402 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:03.572367    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-166f16-28h7j"]
Sep 17 14:33:03.572842 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:03.572825    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3fdb9fc-8cca-4e97-b492-3bf790c235fc" containerName="main"
Sep 17 14:33:03.572892 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:03.572844    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3fdb9fc-8cca-4e97-b492-3bf790c235fc" containerName="main"
Sep 17 14:33:03.572892 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:03.572852    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c03532a3-a9fb-4c4e-af43-8df2c342cad6" containerName="revoke-keys"
Sep 17 14:33:03.572892 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:03.572857    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c03532a3-a9fb-4c4e-af43-8df2c342cad6" containerName="revoke-keys"
Sep 17 14:33:03.572892 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:03.572878    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0" containerName="istio-proxy"
Sep 17 14:33:03.572892 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:03.572884    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0" containerName="istio-proxy"
Sep 17 14:33:03.573052 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:03.572952    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="c03532a3-a9fb-4c4e-af43-8df2c342cad6" containerName="revoke-keys"
Sep 17 14:33:03.573052 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:03.572961    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0" containerName="istio-proxy"
Sep 17 14:33:03.573052 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:03.572967    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3fdb9fc-8cca-4e97-b492-3bf790c235fc" containerName="main"
Sep 17 14:33:03.577453 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:03.577437    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-166f16-28h7j"
Sep 17 14:33:03.585420 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:03.585401    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rp9ww\""
Sep 17 14:33:03.585718 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:03.585698    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 14:33:03.606251 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:03.606225    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-166f16-28h7j"]
Sep 17 14:33:03.637766 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:03.637732    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f6d9668-23ab-4346-aa97-41fe9a007394-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-166f16-28h7j\" (UID: \"2f6d9668-23ab-4346-aa97-41fe9a007394\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-166f16-28h7j"
Sep 17 14:33:03.739219 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:03.739180    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f6d9668-23ab-4346-aa97-41fe9a007394-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-166f16-28h7j\" (UID: \"2f6d9668-23ab-4346-aa97-41fe9a007394\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-166f16-28h7j"
Sep 17 14:33:03.739884 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:03.739855    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f6d9668-23ab-4346-aa97-41fe9a007394-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-166f16-28h7j\" (UID: \"2f6d9668-23ab-4346-aa97-41fe9a007394\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-166f16-28h7j"
Sep 17 14:33:03.887637 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:03.887546    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-166f16-28h7j"
Sep 17 14:33:04.053493 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:04.053443    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:04.174778 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:04.174751    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-166f16-28h7j"]
Sep 17 14:33:04.176505 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:33:04.176479    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f6d9668_23ab_4346_aa97_41fe9a007394.slice/crio-6c31b0a55a7637b32c4daf6ae3320ca3effa4c4a967650fea915e2af0efca1a7 WatchSource:0}: Error finding container 6c31b0a55a7637b32c4daf6ae3320ca3effa4c4a967650fea915e2af0efca1a7: Status 404 returned error can't find the container with id 6c31b0a55a7637b32c4daf6ae3320ca3effa4c4a967650fea915e2af0efca1a7
Sep 17 14:33:04.294657 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:04.294626    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-166f16-28h7j" event={"ID":"2f6d9668-23ab-4346-aa97-41fe9a007394","Type":"ContainerStarted","Data":"6c31b0a55a7637b32c4daf6ae3320ca3effa4c4a967650fea915e2af0efca1a7"}
Sep 17 14:33:04.941257 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:04.941221    2560 patch_prober.go:28] interesting pod/e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 17 14:33:04.941635 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:04.941284    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf" podUID="4633a212-1cff-495c-80df-ceaa196237ae" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 14:33:04.963703 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:04.963666    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:05.302667 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:05.302630    2560 generic.go:356] "Generic (PLEG): container finished" podID="2f6d9668-23ab-4346-aa97-41fe9a007394" containerID="2833fdbb1019d969c8c9338902cea329d6120e56bf01d4c12e7eba9b575873ff" exitCode=0
Sep 17 14:33:05.302842 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:05.302719    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-166f16-28h7j" event={"ID":"2f6d9668-23ab-4346-aa97-41fe9a007394","Type":"ContainerDied","Data":"2833fdbb1019d969c8c9338902cea329d6120e56bf01d4c12e7eba9b575873ff"}
Sep 17 14:33:05.969282 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:05.969246    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:06.441447 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:06.441422    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-166f16-28h7j"
Sep 17 14:33:06.565839 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:06.565763    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f6d9668-23ab-4346-aa97-41fe9a007394-maas-api-service-ca\") pod \"2f6d9668-23ab-4346-aa97-41fe9a007394\" (UID: \"2f6d9668-23ab-4346-aa97-41fe9a007394\") "
Sep 17 14:33:06.566111 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:06.566087    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f6d9668-23ab-4346-aa97-41fe9a007394-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2f6d9668-23ab-4346-aa97-41fe9a007394" (UID: "2f6d9668-23ab-4346-aa97-41fe9a007394"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:33:06.666664 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:06.666635    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f6d9668-23ab-4346-aa97-41fe9a007394-maas-api-service-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:33:06.953395 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:06.953308    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:07.316051 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:07.316024    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-166f16-28h7j"
Sep 17 14:33:07.316638 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:07.316028    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-166f16-28h7j" event={"ID":"2f6d9668-23ab-4346-aa97-41fe9a007394","Type":"ContainerDied","Data":"6c31b0a55a7637b32c4daf6ae3320ca3effa4c4a967650fea915e2af0efca1a7"}
Sep 17 14:33:07.316638 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:07.316137    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c31b0a55a7637b32c4daf6ae3320ca3effa4c4a967650fea915e2af0efca1a7"
Sep 17 14:33:08.006087 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:08.006039    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:08.325387 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:08.325358    2560 generic.go:356] "Generic (PLEG): container finished" podID="4633a212-1cff-495c-80df-ceaa196237ae" containerID="9a9723be0ff9a613fabb68ecebcd627f3e5e896183b0d5e02bc4a601b982ab43" exitCode=0
Sep 17 14:33:08.325760 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:08.325396    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf" event={"ID":"4633a212-1cff-495c-80df-ceaa196237ae","Type":"ContainerDied","Data":"9a9723be0ff9a613fabb68ecebcd627f3e5e896183b0d5e02bc4a601b982ab43"}
Sep 17 14:33:08.325760 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:08.325488    2560 scope.go:117] "RemoveContainer" containerID="bd8ac9dafdf589a3968e1c43dc6f231cf6ac0cc77fdb54e92cd1de2dc998ddf4"
Sep 17 14:33:08.376955 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:08.376934    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf"
Sep 17 14:33:08.485233 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:08.485202    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l6hl4\" (UniqueName: \"kubernetes.io/projected/4633a212-1cff-495c-80df-ceaa196237ae-kube-api-access-l6hl4\") pod \"4633a212-1cff-495c-80df-ceaa196237ae\" (UID: \"4633a212-1cff-495c-80df-ceaa196237ae\") "
Sep 17 14:33:08.485396 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:08.485241    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4633a212-1cff-495c-80df-ceaa196237ae-istio-envoy\") pod \"4633a212-1cff-495c-80df-ceaa196237ae\" (UID: \"4633a212-1cff-495c-80df-ceaa196237ae\") "
Sep 17 14:33:08.485396 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:08.485261    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4633a212-1cff-495c-80df-ceaa196237ae-istio-data\") pod \"4633a212-1cff-495c-80df-ceaa196237ae\" (UID: \"4633a212-1cff-495c-80df-ceaa196237ae\") "
Sep 17 14:33:08.485396 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:08.485318    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4633a212-1cff-495c-80df-ceaa196237ae-workload-socket\") pod \"4633a212-1cff-495c-80df-ceaa196237ae\" (UID: \"4633a212-1cff-495c-80df-ceaa196237ae\") "
Sep 17 14:33:08.485396 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:08.485350    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4633a212-1cff-495c-80df-ceaa196237ae-istiod-ca-cert\") pod \"4633a212-1cff-495c-80df-ceaa196237ae\" (UID: \"4633a212-1cff-495c-80df-ceaa196237ae\") "
Sep 17 14:33:08.485396 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:08.485380    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4633a212-1cff-495c-80df-ceaa196237ae-credential-socket\") pod \"4633a212-1cff-495c-80df-ceaa196237ae\" (UID: \"4633a212-1cff-495c-80df-ceaa196237ae\") "
Sep 17 14:33:08.485697 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:08.485405    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4633a212-1cff-495c-80df-ceaa196237ae-workload-certs\") pod \"4633a212-1cff-495c-80df-ceaa196237ae\" (UID: \"4633a212-1cff-495c-80df-ceaa196237ae\") "
Sep 17 14:33:08.485697 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:08.485443    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4633a212-1cff-495c-80df-ceaa196237ae-istio-token\") pod \"4633a212-1cff-495c-80df-ceaa196237ae\" (UID: \"4633a212-1cff-495c-80df-ceaa196237ae\") "
Sep 17 14:33:08.485697 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:08.485502    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4633a212-1cff-495c-80df-ceaa196237ae-istio-podinfo\") pod \"4633a212-1cff-495c-80df-ceaa196237ae\" (UID: \"4633a212-1cff-495c-80df-ceaa196237ae\") "
Sep 17 14:33:08.485697 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:08.485615    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4633a212-1cff-495c-80df-ceaa196237ae-istio-data" (OuterVolumeSpecName: "istio-data") pod "4633a212-1cff-495c-80df-ceaa196237ae" (UID: "4633a212-1cff-495c-80df-ceaa196237ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:33:08.485697 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:08.485664    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4633a212-1cff-495c-80df-ceaa196237ae-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4633a212-1cff-495c-80df-ceaa196237ae" (UID: "4633a212-1cff-495c-80df-ceaa196237ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:33:08.485923 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:08.485713    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4633a212-1cff-495c-80df-ceaa196237ae-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4633a212-1cff-495c-80df-ceaa196237ae" (UID: "4633a212-1cff-495c-80df-ceaa196237ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:33:08.485923 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:08.485765    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4633a212-1cff-495c-80df-ceaa196237ae-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4633a212-1cff-495c-80df-ceaa196237ae" (UID: "4633a212-1cff-495c-80df-ceaa196237ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:33:08.486045 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:08.485958    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4633a212-1cff-495c-80df-ceaa196237ae-istio-data\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:33:08.486045 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:08.485983    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4633a212-1cff-495c-80df-ceaa196237ae-workload-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:33:08.486045 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:08.485998    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4633a212-1cff-495c-80df-ceaa196237ae-credential-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:33:08.486045 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:08.486012    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4633a212-1cff-495c-80df-ceaa196237ae-workload-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:33:08.486045 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:08.486027    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4633a212-1cff-495c-80df-ceaa196237ae-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4633a212-1cff-495c-80df-ceaa196237ae" (UID: "4633a212-1cff-495c-80df-ceaa196237ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:33:08.487474 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:08.487434    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4633a212-1cff-495c-80df-ceaa196237ae-kube-api-access-l6hl4" (OuterVolumeSpecName: "kube-api-access-l6hl4") pod "4633a212-1cff-495c-80df-ceaa196237ae" (UID: "4633a212-1cff-495c-80df-ceaa196237ae"). InnerVolumeSpecName "kube-api-access-l6hl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:33:08.487587 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:08.487523    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4633a212-1cff-495c-80df-ceaa196237ae-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4633a212-1cff-495c-80df-ceaa196237ae" (UID: "4633a212-1cff-495c-80df-ceaa196237ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:33:08.487587 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:08.487584    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4633a212-1cff-495c-80df-ceaa196237ae-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4633a212-1cff-495c-80df-ceaa196237ae" (UID: "4633a212-1cff-495c-80df-ceaa196237ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 14:33:08.487776 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:08.487755    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4633a212-1cff-495c-80df-ceaa196237ae-istio-token" (OuterVolumeSpecName: "istio-token") pod "4633a212-1cff-495c-80df-ceaa196237ae" (UID: "4633a212-1cff-495c-80df-ceaa196237ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:33:08.587488 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:08.587438    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4633a212-1cff-495c-80df-ceaa196237ae-istiod-ca-cert\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:33:08.587488 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:08.587489    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4633a212-1cff-495c-80df-ceaa196237ae-istio-token\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:33:08.587488 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:08.587500    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4633a212-1cff-495c-80df-ceaa196237ae-istio-podinfo\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:33:08.587679 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:08.587508    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6hl4\" (UniqueName: \"kubernetes.io/projected/4633a212-1cff-495c-80df-ceaa196237ae-kube-api-access-l6hl4\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:33:08.587679 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:08.587517    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4633a212-1cff-495c-80df-ceaa196237ae-istio-envoy\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:33:09.333002 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:09.332971    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf"
Sep 17 14:33:09.333408 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:09.333014    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf" event={"ID":"4633a212-1cff-495c-80df-ceaa196237ae","Type":"ContainerDied","Data":"cc3b4b9f6fe813bed0d3b5a79ebac354bb31f919b215f1cfe93129888eada71b"}
Sep 17 14:33:09.333408 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:09.333072    2560 scope.go:117] "RemoveContainer" containerID="9a9723be0ff9a613fabb68ecebcd627f3e5e896183b0d5e02bc4a601b982ab43"
Sep 17 14:33:09.357145 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:09.357120    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf"]
Sep 17 14:33:09.360180 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:09.360154    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-911a82-openshift-default-55764bfb98-52mmf"]
Sep 17 14:33:09.569884 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:09.569852    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4633a212-1cff-495c-80df-ceaa196237ae" path="/var/lib/kubelet/pods/4633a212-1cff-495c-80df-ceaa196237ae/volumes"
Sep 17 14:33:14.271043 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:14.271011    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:14.360315 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:14.360284    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:14.428441 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:14.428406    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:14.528851 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:14.528763    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:14.538822 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:14.538785    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-166f16-28h7j"]
Sep 17 14:33:14.541474 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:14.541442    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-166f16-28h7j"]
Sep 17 14:33:14.823973 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:14.823893    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:14.974014 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:14.973978    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:15.083765 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:15.083671    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:15.569663 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:15.569630    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f6d9668-23ab-4346-aa97-41fe9a007394" path="/var/lib/kubelet/pods/2f6d9668-23ab-4346-aa97-41fe9a007394/volumes"
Sep 17 14:33:16.032650 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:16.032574    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:16.824781 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:16.824743    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:21.158648 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:21.158617    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:21.234317 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:21.234281    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:21.404859 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:21.404822    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg"]
Sep 17 14:33:21.405086 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:21.405062    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg" podUID="b7c163b0-e401-4279-aaa2-2def6ac2f74b" containerName="istio-proxy" containerID="cri-o://5c988c9123e5357e7bc715427e2e95d977e6ad2abfc0865ee0d2d4c19735eb4f" gracePeriod=30
Sep 17 14:33:21.423086 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:21.423021    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:21.791248 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:21.791156    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b88b90-grhkv"]
Sep 17 14:33:21.791681 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:21.791666    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0" containerName="istio-proxy"
Sep 17 14:33:21.791741 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:21.791683    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0" containerName="istio-proxy"
Sep 17 14:33:21.791741 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:21.791691    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4633a212-1cff-495c-80df-ceaa196237ae" containerName="istio-proxy"
Sep 17 14:33:21.791741 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:21.791697    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4633a212-1cff-495c-80df-ceaa196237ae" containerName="istio-proxy"
Sep 17 14:33:21.791741 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:21.791713    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f6d9668-23ab-4346-aa97-41fe9a007394" containerName="revoke-keys"
Sep 17 14:33:21.791741 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:21.791718    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f6d9668-23ab-4346-aa97-41fe9a007394" containerName="revoke-keys"
Sep 17 14:33:21.791741 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:21.791729    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4633a212-1cff-495c-80df-ceaa196237ae" containerName="istio-proxy"
Sep 17 14:33:21.791741 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:21.791734    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4633a212-1cff-495c-80df-ceaa196237ae" containerName="istio-proxy"
Sep 17 14:33:21.791970 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:21.791800    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="2405ae78-bb3d-45c7-8bdb-a29fe98cd8d0" containerName="istio-proxy"
Sep 17 14:33:21.791970 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:21.791810    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f6d9668-23ab-4346-aa97-41fe9a007394" containerName="revoke-keys"
Sep 17 14:33:21.791970 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:21.791817    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4633a212-1cff-495c-80df-ceaa196237ae" containerName="istio-proxy"
Sep 17 14:33:21.791970 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:21.791823    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4633a212-1cff-495c-80df-ceaa196237ae" containerName="istio-proxy"
Sep 17 14:33:21.796363 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:21.796345    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b88b90-grhkv"
Sep 17 14:33:21.798589 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:21.798568    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 14:33:21.798693 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:21.798636    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rp9ww\""
Sep 17 14:33:21.808018 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:21.807995    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b88b90-grhkv"]
Sep 17 14:33:21.821529 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:21.821507    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be392fcb-8cc3-403f-b184-c26b1cad18aa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-b88b90-grhkv\" (UID: \"be392fcb-8cc3-403f-b184-c26b1cad18aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b88b90-grhkv"
Sep 17 14:33:21.922128 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:21.922103    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be392fcb-8cc3-403f-b184-c26b1cad18aa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-b88b90-grhkv\" (UID: \"be392fcb-8cc3-403f-b184-c26b1cad18aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b88b90-grhkv"
Sep 17 14:33:21.922734 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:21.922716    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be392fcb-8cc3-403f-b184-c26b1cad18aa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-b88b90-grhkv\" (UID: \"be392fcb-8cc3-403f-b184-c26b1cad18aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b88b90-grhkv"
Sep 17 14:33:21.968848 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:21.968821    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:22.107181 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:22.107133    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b88b90-grhkv"
Sep 17 14:33:22.382526 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:22.382497    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b88b90-grhkv"]
Sep 17 14:33:22.384126 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:33:22.384100    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe392fcb_8cc3_403f_b184_c26b1cad18aa.slice/crio-8d993853d6ec9cd74dde7817b2a38d00e56b94554126ffe244a55d2ff2cd1740 WatchSource:0}: Error finding container 8d993853d6ec9cd74dde7817b2a38d00e56b94554126ffe244a55d2ff2cd1740: Status 404 returned error can't find the container with id 8d993853d6ec9cd74dde7817b2a38d00e56b94554126ffe244a55d2ff2cd1740
Sep 17 14:33:22.415830 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:22.415803    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b88b90-grhkv" event={"ID":"be392fcb-8cc3-403f-b184-c26b1cad18aa","Type":"ContainerStarted","Data":"8d993853d6ec9cd74dde7817b2a38d00e56b94554126ffe244a55d2ff2cd1740"}
Sep 17 14:33:22.765024 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:22.764984    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:23.423343 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:23.423308    2560 generic.go:356] "Generic (PLEG): container finished" podID="be392fcb-8cc3-403f-b184-c26b1cad18aa" containerID="f6c17f1e82d1682f13f3c2e0f1ae3f2ab118fcae8646197e6bcd8d43e734e0a3" exitCode=0
Sep 17 14:33:23.423796 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:23.423360    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b88b90-grhkv" event={"ID":"be392fcb-8cc3-403f-b184-c26b1cad18aa","Type":"ContainerDied","Data":"f6c17f1e82d1682f13f3c2e0f1ae3f2ab118fcae8646197e6bcd8d43e734e0a3"}
Sep 17 14:33:23.557705 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:23.557654    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:24.564151 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:24.564121    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b88b90-grhkv"
Sep 17 14:33:24.642886 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:24.642853    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:24.649992 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:24.649968    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be392fcb-8cc3-403f-b184-c26b1cad18aa-maas-api-service-ca\") pod \"be392fcb-8cc3-403f-b184-c26b1cad18aa\" (UID: \"be392fcb-8cc3-403f-b184-c26b1cad18aa\") "
Sep 17 14:33:24.650299 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:24.650279    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be392fcb-8cc3-403f-b184-c26b1cad18aa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "be392fcb-8cc3-403f-b184-c26b1cad18aa" (UID: "be392fcb-8cc3-403f-b184-c26b1cad18aa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:33:24.751393 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:24.751313    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be392fcb-8cc3-403f-b184-c26b1cad18aa-maas-api-service-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:33:25.410799 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:25.410770    2560 patch_prober.go:28] interesting pod/e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 17 14:33:25.410955 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:25.410815    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg" podUID="b7c163b0-e401-4279-aaa2-2def6ac2f74b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 14:33:25.437136 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:25.437108    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b88b90-grhkv"
Sep 17 14:33:25.437263 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:25.437139    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b88b90-grhkv" event={"ID":"be392fcb-8cc3-403f-b184-c26b1cad18aa","Type":"ContainerDied","Data":"8d993853d6ec9cd74dde7817b2a38d00e56b94554126ffe244a55d2ff2cd1740"}
Sep 17 14:33:25.437263 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:25.437188    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d993853d6ec9cd74dde7817b2a38d00e56b94554126ffe244a55d2ff2cd1740"
Sep 17 14:33:26.660740 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:26.660713    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg"
Sep 17 14:33:26.771506 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:26.771408    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7c163b0-e401-4279-aaa2-2def6ac2f74b-istio-token\") pod \"b7c163b0-e401-4279-aaa2-2def6ac2f74b\" (UID: \"b7c163b0-e401-4279-aaa2-2def6ac2f74b\") "
Sep 17 14:33:26.771506 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:26.771448    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7c163b0-e401-4279-aaa2-2def6ac2f74b-istio-podinfo\") pod \"b7c163b0-e401-4279-aaa2-2def6ac2f74b\" (UID: \"b7c163b0-e401-4279-aaa2-2def6ac2f74b\") "
Sep 17 14:33:26.771735 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:26.771521    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7c163b0-e401-4279-aaa2-2def6ac2f74b-istio-data\") pod \"b7c163b0-e401-4279-aaa2-2def6ac2f74b\" (UID: \"b7c163b0-e401-4279-aaa2-2def6ac2f74b\") "
Sep 17 14:33:26.771735 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:26.771540    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7zr2m\" (UniqueName: \"kubernetes.io/projected/b7c163b0-e401-4279-aaa2-2def6ac2f74b-kube-api-access-7zr2m\") pod \"b7c163b0-e401-4279-aaa2-2def6ac2f74b\" (UID: \"b7c163b0-e401-4279-aaa2-2def6ac2f74b\") "
Sep 17 14:33:26.771735 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:26.771560    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7c163b0-e401-4279-aaa2-2def6ac2f74b-credential-socket\") pod \"b7c163b0-e401-4279-aaa2-2def6ac2f74b\" (UID: \"b7c163b0-e401-4279-aaa2-2def6ac2f74b\") "
Sep 17 14:33:26.771735 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:26.771700    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7c163b0-e401-4279-aaa2-2def6ac2f74b-workload-socket\") pod \"b7c163b0-e401-4279-aaa2-2def6ac2f74b\" (UID: \"b7c163b0-e401-4279-aaa2-2def6ac2f74b\") "
Sep 17 14:33:26.771947 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:26.771813    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7c163b0-e401-4279-aaa2-2def6ac2f74b-workload-certs\") pod \"b7c163b0-e401-4279-aaa2-2def6ac2f74b\" (UID: \"b7c163b0-e401-4279-aaa2-2def6ac2f74b\") "
Sep 17 14:33:26.771947 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:26.771840    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7c163b0-e401-4279-aaa2-2def6ac2f74b-istio-envoy\") pod \"b7c163b0-e401-4279-aaa2-2def6ac2f74b\" (UID: \"b7c163b0-e401-4279-aaa2-2def6ac2f74b\") "
Sep 17 14:33:26.771947 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:26.771867    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7c163b0-e401-4279-aaa2-2def6ac2f74b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b7c163b0-e401-4279-aaa2-2def6ac2f74b" (UID: "b7c163b0-e401-4279-aaa2-2def6ac2f74b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:33:26.771947 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:26.771876    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7c163b0-e401-4279-aaa2-2def6ac2f74b-istiod-ca-cert\") pod \"b7c163b0-e401-4279-aaa2-2def6ac2f74b\" (UID: \"b7c163b0-e401-4279-aaa2-2def6ac2f74b\") "
Sep 17 14:33:26.771947 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:26.771877    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7c163b0-e401-4279-aaa2-2def6ac2f74b-istio-data" (OuterVolumeSpecName: "istio-data") pod "b7c163b0-e401-4279-aaa2-2def6ac2f74b" (UID: "b7c163b0-e401-4279-aaa2-2def6ac2f74b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:33:26.772218 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:26.772054    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7c163b0-e401-4279-aaa2-2def6ac2f74b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b7c163b0-e401-4279-aaa2-2def6ac2f74b" (UID: "b7c163b0-e401-4279-aaa2-2def6ac2f74b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:33:26.772278 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:26.772249    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7c163b0-e401-4279-aaa2-2def6ac2f74b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b7c163b0-e401-4279-aaa2-2def6ac2f74b" (UID: "b7c163b0-e401-4279-aaa2-2def6ac2f74b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:33:26.772352 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:26.772330    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7c163b0-e401-4279-aaa2-2def6ac2f74b-istio-data\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:33:26.772414 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:26.772357    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7c163b0-e401-4279-aaa2-2def6ac2f74b-credential-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:33:26.772414 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:26.772374    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7c163b0-e401-4279-aaa2-2def6ac2f74b-workload-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:33:26.772555 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:26.772494    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7c163b0-e401-4279-aaa2-2def6ac2f74b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b7c163b0-e401-4279-aaa2-2def6ac2f74b" (UID: "b7c163b0-e401-4279-aaa2-2def6ac2f74b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:33:26.773802 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:26.773774    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7c163b0-e401-4279-aaa2-2def6ac2f74b-istio-token" (OuterVolumeSpecName: "istio-token") pod "b7c163b0-e401-4279-aaa2-2def6ac2f74b" (UID: "b7c163b0-e401-4279-aaa2-2def6ac2f74b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:33:26.773909 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:26.773843    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7c163b0-e401-4279-aaa2-2def6ac2f74b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b7c163b0-e401-4279-aaa2-2def6ac2f74b" (UID: "b7c163b0-e401-4279-aaa2-2def6ac2f74b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:33:26.773909 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:26.773890    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b7c163b0-e401-4279-aaa2-2def6ac2f74b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b7c163b0-e401-4279-aaa2-2def6ac2f74b" (UID: "b7c163b0-e401-4279-aaa2-2def6ac2f74b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 14:33:26.773909 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:26.773897    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7c163b0-e401-4279-aaa2-2def6ac2f74b-kube-api-access-7zr2m" (OuterVolumeSpecName: "kube-api-access-7zr2m") pod "b7c163b0-e401-4279-aaa2-2def6ac2f74b" (UID: "b7c163b0-e401-4279-aaa2-2def6ac2f74b"). InnerVolumeSpecName "kube-api-access-7zr2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:33:26.872926 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:26.872900    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7c163b0-e401-4279-aaa2-2def6ac2f74b-istio-envoy\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:33:26.872926 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:26.872923    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7c163b0-e401-4279-aaa2-2def6ac2f74b-istiod-ca-cert\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:33:26.872926 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:26.872931    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7c163b0-e401-4279-aaa2-2def6ac2f74b-istio-token\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:33:26.873101 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:26.872939    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7c163b0-e401-4279-aaa2-2def6ac2f74b-istio-podinfo\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:33:26.873101 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:26.872948    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zr2m\" (UniqueName: \"kubernetes.io/projected/b7c163b0-e401-4279-aaa2-2def6ac2f74b-kube-api-access-7zr2m\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:33:26.873101 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:26.872956    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7c163b0-e401-4279-aaa2-2def6ac2f74b-workload-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:33:27.450010 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:27.449975    2560 generic.go:356] "Generic (PLEG): container finished" podID="b7c163b0-e401-4279-aaa2-2def6ac2f74b" containerID="5c988c9123e5357e7bc715427e2e95d977e6ad2abfc0865ee0d2d4c19735eb4f" exitCode=0
Sep 17 14:33:27.450181 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:27.450030    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg"
Sep 17 14:33:27.450181 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:27.450059    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg" event={"ID":"b7c163b0-e401-4279-aaa2-2def6ac2f74b","Type":"ContainerDied","Data":"5c988c9123e5357e7bc715427e2e95d977e6ad2abfc0865ee0d2d4c19735eb4f"}
Sep 17 14:33:27.450181 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:27.450110    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg" event={"ID":"b7c163b0-e401-4279-aaa2-2def6ac2f74b","Type":"ContainerDied","Data":"d266440425bd4122da2cdf320eba6c7a3809f5b391c4a4fc47b3c69e34bb181a"}
Sep 17 14:33:27.450181 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:27.450133    2560 scope.go:117] "RemoveContainer" containerID="5c988c9123e5357e7bc715427e2e95d977e6ad2abfc0865ee0d2d4c19735eb4f"
Sep 17 14:33:27.460733 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:27.460716    2560 scope.go:117] "RemoveContainer" containerID="27bbfc02491f40cd2ed63dc321ce125db24e4212360a123e18b23da49d32d4b1"
Sep 17 14:33:27.468992 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:27.468978    2560 scope.go:117] "RemoveContainer" containerID="5c988c9123e5357e7bc715427e2e95d977e6ad2abfc0865ee0d2d4c19735eb4f"
Sep 17 14:33:27.469245 ip-10-0-129-158 kubenswrapper[2560]: E0917 14:33:27.469228    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c988c9123e5357e7bc715427e2e95d977e6ad2abfc0865ee0d2d4c19735eb4f\": container with ID starting with 5c988c9123e5357e7bc715427e2e95d977e6ad2abfc0865ee0d2d4c19735eb4f not found: ID does not exist" containerID="5c988c9123e5357e7bc715427e2e95d977e6ad2abfc0865ee0d2d4c19735eb4f"
Sep 17 14:33:27.469291 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:27.469252    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c988c9123e5357e7bc715427e2e95d977e6ad2abfc0865ee0d2d4c19735eb4f"} err="failed to get container status \"5c988c9123e5357e7bc715427e2e95d977e6ad2abfc0865ee0d2d4c19735eb4f\": rpc error: code = NotFound desc = could not find container \"5c988c9123e5357e7bc715427e2e95d977e6ad2abfc0865ee0d2d4c19735eb4f\": container with ID starting with 5c988c9123e5357e7bc715427e2e95d977e6ad2abfc0865ee0d2d4c19735eb4f not found: ID does not exist"
Sep 17 14:33:27.469291 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:27.469268    2560 scope.go:117] "RemoveContainer" containerID="27bbfc02491f40cd2ed63dc321ce125db24e4212360a123e18b23da49d32d4b1"
Sep 17 14:33:27.469531 ip-10-0-129-158 kubenswrapper[2560]: E0917 14:33:27.469514    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27bbfc02491f40cd2ed63dc321ce125db24e4212360a123e18b23da49d32d4b1\": container with ID starting with 27bbfc02491f40cd2ed63dc321ce125db24e4212360a123e18b23da49d32d4b1 not found: ID does not exist" containerID="27bbfc02491f40cd2ed63dc321ce125db24e4212360a123e18b23da49d32d4b1"
Sep 17 14:33:27.469588 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:27.469542    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27bbfc02491f40cd2ed63dc321ce125db24e4212360a123e18b23da49d32d4b1"} err="failed to get container status \"27bbfc02491f40cd2ed63dc321ce125db24e4212360a123e18b23da49d32d4b1\": rpc error: code = NotFound desc = could not find container \"27bbfc02491f40cd2ed63dc321ce125db24e4212360a123e18b23da49d32d4b1\": container with ID starting with 27bbfc02491f40cd2ed63dc321ce125db24e4212360a123e18b23da49d32d4b1 not found: ID does not exist"
Sep 17 14:33:27.477714 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:27.477692    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg"]
Sep 17 14:33:27.490097 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:27.490078    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-166f16-openshift-default-6f8758d8d4-wdvbg"]
Sep 17 14:33:27.569333 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:27.569305    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7c163b0-e401-4279-aaa2-2def6ac2f74b" path="/var/lib/kubelet/pods/b7c163b0-e401-4279-aaa2-2def6ac2f74b/volumes"
Sep 17 14:33:32.503298 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:32.503257    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:32.510724 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:32.510689    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b88b90-29827590-ngjln"]
Sep 17 14:33:32.513325 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:32.513297    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b88b90-29827590-ngjln"]
Sep 17 14:33:32.587983 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:32.587932    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:32.651368 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:32.651330    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:32.751377 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:32.751332    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:32.781131 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:32.781100    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b88b90-grhkv"]
Sep 17 14:33:32.783581 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:32.783557    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b88b90-grhkv"]
Sep 17 14:33:32.823508 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:32.823441    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:33.163951 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:33.163913    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:33.265888 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:33.265839    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:33.569476 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:33.569421    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94f0a0b6-9286-421c-8c15-c382f63dbef2" path="/var/lib/kubelet/pods/94f0a0b6-9286-421c-8c15-c382f63dbef2/volumes"
Sep 17 14:33:33.569858 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:33.569791    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be392fcb-8cc3-403f-b184-c26b1cad18aa" path="/var/lib/kubelet/pods/be392fcb-8cc3-403f-b184-c26b1cad18aa/volumes"
Sep 17 14:33:34.407086 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:34.407030    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:35.955251 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:35.955213    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:36.017003 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:36.016966    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:40.033192 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:40.033155    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:40.133863 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:40.133826    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:40.303781 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:40.303747    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:40.499338 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:40.499305    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:40.661589 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:40.661504    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b79988-g56q8"]
Sep 17 14:33:40.661913 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:40.661900    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7c163b0-e401-4279-aaa2-2def6ac2f74b" containerName="istio-proxy"
Sep 17 14:33:40.661956 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:40.661914    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7c163b0-e401-4279-aaa2-2def6ac2f74b" containerName="istio-proxy"
Sep 17 14:33:40.661956 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:40.661924    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7c163b0-e401-4279-aaa2-2def6ac2f74b" containerName="istio-proxy"
Sep 17 14:33:40.661956 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:40.661931    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7c163b0-e401-4279-aaa2-2def6ac2f74b" containerName="istio-proxy"
Sep 17 14:33:40.661956 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:40.661950    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be392fcb-8cc3-403f-b184-c26b1cad18aa" containerName="revoke-keys"
Sep 17 14:33:40.661956 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:40.661955    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="be392fcb-8cc3-403f-b184-c26b1cad18aa" containerName="revoke-keys"
Sep 17 14:33:40.662121 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:40.662025    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7c163b0-e401-4279-aaa2-2def6ac2f74b" containerName="istio-proxy"
Sep 17 14:33:40.662121 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:40.662031    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="be392fcb-8cc3-403f-b184-c26b1cad18aa" containerName="revoke-keys"
Sep 17 14:33:40.662121 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:40.662039    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7c163b0-e401-4279-aaa2-2def6ac2f74b" containerName="istio-proxy"
Sep 17 14:33:40.670894 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:40.670859    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b79988-g56q8"
Sep 17 14:33:40.673341 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:40.673291    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 14:33:40.673580 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:40.673402    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rp9ww\""
Sep 17 14:33:40.675781 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:40.675754    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b79988-g56q8"]
Sep 17 14:33:40.809697 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:40.809656    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd372b4e-b3be-448a-a978-bf83445061fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-b79988-g56q8\" (UID: \"cd372b4e-b3be-448a-a978-bf83445061fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b79988-g56q8"
Sep 17 14:33:40.911253 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:40.911212    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd372b4e-b3be-448a-a978-bf83445061fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-b79988-g56q8\" (UID: \"cd372b4e-b3be-448a-a978-bf83445061fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b79988-g56q8"
Sep 17 14:33:40.912002 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:40.911947    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd372b4e-b3be-448a-a978-bf83445061fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-b79988-g56q8\" (UID: \"cd372b4e-b3be-448a-a978-bf83445061fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b79988-g56q8"
Sep 17 14:33:40.964649 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:40.964613    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:40.984332 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:40.984304    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b79988-g56q8"
Sep 17 14:33:41.261554 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:41.261525    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b79988-g56q8"]
Sep 17 14:33:41.263383 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:33:41.263354    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd372b4e_b3be_448a_a978_bf83445061fd.slice/crio-1ae58d9c9581e68aece1cd70f3c59246eaa4a971d9221f9de15ef186e696d916 WatchSource:0}: Error finding container 1ae58d9c9581e68aece1cd70f3c59246eaa4a971d9221f9de15ef186e696d916: Status 404 returned error can't find the container with id 1ae58d9c9581e68aece1cd70f3c59246eaa4a971d9221f9de15ef186e696d916
Sep 17 14:33:41.538216 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:41.538127    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b79988-g56q8" event={"ID":"cd372b4e-b3be-448a-a978-bf83445061fd","Type":"ContainerStarted","Data":"1ae58d9c9581e68aece1cd70f3c59246eaa4a971d9221f9de15ef186e696d916"}
Sep 17 14:33:41.743488 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:41.743428    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:42.546690 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:42.546656    2560 generic.go:356] "Generic (PLEG): container finished" podID="cd372b4e-b3be-448a-a978-bf83445061fd" containerID="f8cd452647130dbeafe9dc7537ce88fd525de97456fb24bb6a2ec4b0d65c9f32" exitCode=0
Sep 17 14:33:42.547115 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:42.546754    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b79988-g56q8" event={"ID":"cd372b4e-b3be-448a-a978-bf83445061fd","Type":"ContainerDied","Data":"f8cd452647130dbeafe9dc7537ce88fd525de97456fb24bb6a2ec4b0d65c9f32"}
Sep 17 14:33:42.647562 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:42.647506    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:43.685239 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:43.685216    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b79988-g56q8"
Sep 17 14:33:43.839966 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:43.839882    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd372b4e-b3be-448a-a978-bf83445061fd-maas-api-service-ca\") pod \"cd372b4e-b3be-448a-a978-bf83445061fd\" (UID: \"cd372b4e-b3be-448a-a978-bf83445061fd\") "
Sep 17 14:33:43.840279 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:43.840257    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd372b4e-b3be-448a-a978-bf83445061fd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cd372b4e-b3be-448a-a978-bf83445061fd" (UID: "cd372b4e-b3be-448a-a978-bf83445061fd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:33:43.940870 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:43.940839    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd372b4e-b3be-448a-a978-bf83445061fd-maas-api-service-ca\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:33:44.560069 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:44.560034    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b79988-g56q8" event={"ID":"cd372b4e-b3be-448a-a978-bf83445061fd","Type":"ContainerDied","Data":"1ae58d9c9581e68aece1cd70f3c59246eaa4a971d9221f9de15ef186e696d916"}
Sep 17 14:33:44.560069 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:44.560069    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b79988-g56q8"
Sep 17 14:33:44.560299 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:44.560086    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ae58d9c9581e68aece1cd70f3c59246eaa4a971d9221f9de15ef186e696d916"
Sep 17 14:33:51.373744 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:51.373710    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:51.470314 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:51.470282    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:51.570352 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:51.570315    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:51.594115 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:51.594090    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b79988-g56q8"]
Sep 17 14:33:51.596601 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:51.596580    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b79988-g56q8"]
Sep 17 14:33:51.636558 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:51.636484    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:51.965841 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:51.965745    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:52.066539 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:52.066486    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:53.569874 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:53.569843    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd372b4e-b3be-448a-a978-bf83445061fd" path="/var/lib/kubelet/pods/cd372b4e-b3be-448a-a978-bf83445061fd/volumes"
Sep 17 14:33:59.352054 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:59.352013    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:59.455565 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:59.455510    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:59.637643 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:59.637554    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk"]
Sep 17 14:33:59.637808 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:59.637772    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk" podUID="00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c" containerName="istio-proxy" containerID="cri-o://37df4fdacbf70607e486aa4270c16db4ff2fba4c42bc6c2034ecc83a72fbe048" gracePeriod=30
Sep 17 14:33:59.670946 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:33:59.670913    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:04.896345 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:34:04.896319    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk"
Sep 17 14:34:04.942178 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:34:04.942090    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c-workload-certs\") pod \"00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c\" (UID: \"00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c\") "
Sep 17 14:34:04.942178 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:34:04.942141    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5xkjf\" (UniqueName: \"kubernetes.io/projected/00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c-kube-api-access-5xkjf\") pod \"00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c\" (UID: \"00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c\") "
Sep 17 14:34:04.942178 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:34:04.942162    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c-istio-token\") pod \"00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c\" (UID: \"00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c\") "
Sep 17 14:34:04.942498 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:34:04.942196    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c-istio-podinfo\") pod \"00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c\" (UID: \"00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c\") "
Sep 17 14:34:04.942498 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:34:04.942250    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c-istio-envoy\") pod \"00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c\" (UID: \"00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c\") "
Sep 17 14:34:04.942498 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:34:04.942272    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c-workload-socket\") pod \"00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c\" (UID: \"00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c\") "
Sep 17 14:34:04.942498 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:34:04.942300    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c-istiod-ca-cert\") pod \"00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c\" (UID: \"00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c\") "
Sep 17 14:34:04.942498 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:34:04.942340    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c-credential-socket\") pod \"00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c\" (UID: \"00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c\") "
Sep 17 14:34:04.942498 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:34:04.942342    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c" (UID: "00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:34:04.942498 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:34:04.942368    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c-istio-data\") pod \"00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c\" (UID: \"00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c\") "
Sep 17 14:34:04.942890 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:34:04.942641    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c-workload-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:34:04.942890 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:34:04.942674    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c" (UID: "00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:34:04.942890 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:34:04.942779    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c" (UID: "00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:34:04.942890 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:34:04.942810    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c-istio-data" (OuterVolumeSpecName: "istio-data") pod "00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c" (UID: "00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:34:04.943107 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:34:04.942943    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c" (UID: "00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:34:04.944831 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:34:04.944777    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c" (UID: "00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 14:34:04.944963 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:34:04.944830    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c-istio-token" (OuterVolumeSpecName: "istio-token") pod "00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c" (UID: "00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:34:04.944963 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:34:04.944884    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c" (UID: "00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:34:04.944963 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:34:04.944916    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c-kube-api-access-5xkjf" (OuterVolumeSpecName: "kube-api-access-5xkjf") pod "00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c" (UID: "00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c"). InnerVolumeSpecName "kube-api-access-5xkjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:34:05.043871 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:34:05.043824    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xkjf\" (UniqueName: \"kubernetes.io/projected/00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c-kube-api-access-5xkjf\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:34:05.043871 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:34:05.043860    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c-istio-token\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:34:05.043871 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:34:05.043869    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c-istio-podinfo\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:34:05.043871 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:34:05.043879    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c-istio-envoy\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:34:05.044146 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:34:05.043888    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c-workload-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:34:05.044146 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:34:05.043896    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c-istiod-ca-cert\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:34:05.044146 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:34:05.043904    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c-credential-socket\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:34:05.044146 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:34:05.043912    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c-istio-data\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:34:05.690094 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:34:05.690053    2560 generic.go:356] "Generic (PLEG): container finished" podID="00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c" containerID="37df4fdacbf70607e486aa4270c16db4ff2fba4c42bc6c2034ecc83a72fbe048" exitCode=0
Sep 17 14:34:05.690257 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:34:05.690106    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk"
Sep 17 14:34:05.690257 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:34:05.690129    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk" event={"ID":"00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c","Type":"ContainerDied","Data":"37df4fdacbf70607e486aa4270c16db4ff2fba4c42bc6c2034ecc83a72fbe048"}
Sep 17 14:34:05.690257 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:34:05.690186    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk" event={"ID":"00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c","Type":"ContainerDied","Data":"43d1f1e6deae43511776d607572270feb2ac99333fa41599b0a158b8bf8c6676"}
Sep 17 14:34:05.690257 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:34:05.690209    2560 scope.go:117] "RemoveContainer" containerID="37df4fdacbf70607e486aa4270c16db4ff2fba4c42bc6c2034ecc83a72fbe048"
Sep 17 14:34:05.699668 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:34:05.699648    2560 scope.go:117] "RemoveContainer" containerID="e4ab3d7ece22c3f9c65ee25db7fa7a7d68653943f3699646e85483f25740a40b"
Sep 17 14:34:05.708666 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:34:05.708644    2560 scope.go:117] "RemoveContainer" containerID="37df4fdacbf70607e486aa4270c16db4ff2fba4c42bc6c2034ecc83a72fbe048"
Sep 17 14:34:05.708962 ip-10-0-129-158 kubenswrapper[2560]: E0917 14:34:05.708938    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37df4fdacbf70607e486aa4270c16db4ff2fba4c42bc6c2034ecc83a72fbe048\": container with ID starting with 37df4fdacbf70607e486aa4270c16db4ff2fba4c42bc6c2034ecc83a72fbe048 not found: ID does not exist" containerID="37df4fdacbf70607e486aa4270c16db4ff2fba4c42bc6c2034ecc83a72fbe048"
Sep 17 14:34:05.709062 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:34:05.708970    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37df4fdacbf70607e486aa4270c16db4ff2fba4c42bc6c2034ecc83a72fbe048"} err="failed to get container status \"37df4fdacbf70607e486aa4270c16db4ff2fba4c42bc6c2034ecc83a72fbe048\": rpc error: code = NotFound desc = could not find container \"37df4fdacbf70607e486aa4270c16db4ff2fba4c42bc6c2034ecc83a72fbe048\": container with ID starting with 37df4fdacbf70607e486aa4270c16db4ff2fba4c42bc6c2034ecc83a72fbe048 not found: ID does not exist"
Sep 17 14:34:05.709062 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:34:05.708988    2560 scope.go:117] "RemoveContainer" containerID="e4ab3d7ece22c3f9c65ee25db7fa7a7d68653943f3699646e85483f25740a40b"
Sep 17 14:34:05.709283 ip-10-0-129-158 kubenswrapper[2560]: E0917 14:34:05.709261    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4ab3d7ece22c3f9c65ee25db7fa7a7d68653943f3699646e85483f25740a40b\": container with ID starting with e4ab3d7ece22c3f9c65ee25db7fa7a7d68653943f3699646e85483f25740a40b not found: ID does not exist" containerID="e4ab3d7ece22c3f9c65ee25db7fa7a7d68653943f3699646e85483f25740a40b"
Sep 17 14:34:05.709326 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:34:05.709292    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4ab3d7ece22c3f9c65ee25db7fa7a7d68653943f3699646e85483f25740a40b"} err="failed to get container status \"e4ab3d7ece22c3f9c65ee25db7fa7a7d68653943f3699646e85483f25740a40b\": rpc error: code = NotFound desc = could not find container \"e4ab3d7ece22c3f9c65ee25db7fa7a7d68653943f3699646e85483f25740a40b\": container with ID starting with e4ab3d7ece22c3f9c65ee25db7fa7a7d68653943f3699646e85483f25740a40b not found: ID does not exist"
Sep 17 14:34:05.711022 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:34:05.711003    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk"]
Sep 17 14:34:05.716652 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:34:05.716627    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-b79988-openshift-default-5cb7b5cfb4-6wgdk"]
Sep 17 14:34:07.311123 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:34:07.311088    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:07.570312 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:34:07.570233    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c" path="/var/lib/kubelet/pods/00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c/volumes"
Sep 17 14:35:13.878664 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:35:13.878624    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:26.880517 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:35:26.880481    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:28.332643 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:35:28.332611    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:40.054533 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:35:40.054495    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:49.242677 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:35:49.242633    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:36:32.340185 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:36:32.340146    2560 scope.go:117] "RemoveContainer" containerID="91860f2edd354e4cb975af254ee49f900d7689c42c3f447782a1cd8212bb7cbc"
Sep 17 14:38:22.502948 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:38:22.502915    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:38:27.597607 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:38:27.597571    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-d3c310b5-kserve-5c98bfdf87-6sm2w"]
Sep 17 14:38:27.597996 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:38:27.597962    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c" containerName="istio-proxy"
Sep 17 14:38:27.597996 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:38:27.597975    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c" containerName="istio-proxy"
Sep 17 14:38:27.597996 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:38:27.597989    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c" containerName="istio-proxy"
Sep 17 14:38:27.597996 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:38:27.597995    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c" containerName="istio-proxy"
Sep 17 14:38:27.598131 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:38:27.598004    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd372b4e-b3be-448a-a978-bf83445061fd" containerName="revoke-keys"
Sep 17 14:38:27.598131 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:38:27.598010    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd372b4e-b3be-448a-a978-bf83445061fd" containerName="revoke-keys"
Sep 17 14:38:27.598131 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:38:27.598081    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c" containerName="istio-proxy"
Sep 17 14:38:27.598131 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:38:27.598089    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd372b4e-b3be-448a-a978-bf83445061fd" containerName="revoke-keys"
Sep 17 14:38:27.601189 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:38:27.601172    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-d3c310b5-kserve-5c98bfdf87-6sm2w"
Sep 17 14:38:27.603563 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:38:27.603543    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-d3c310b5-kserve-self-signed-certs\""
Sep 17 14:38:27.611269 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:38:27.611247    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-d3c310b5-kserve-5c98bfdf87-6sm2w"]
Sep 17 14:38:27.688409 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:38:27.688367    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlxkc\" (UniqueName: \"kubernetes.io/projected/bb4ed0f4-d684-42c4-9bf2-295f7250e7ae-kube-api-access-tlxkc\") pod \"e2e-same-id-b-d3c310b5-kserve-5c98bfdf87-6sm2w\" (UID: \"bb4ed0f4-d684-42c4-9bf2-295f7250e7ae\") " pod="llm/e2e-same-id-b-d3c310b5-kserve-5c98bfdf87-6sm2w"
Sep 17 14:38:27.688606 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:38:27.688445    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb4ed0f4-d684-42c4-9bf2-295f7250e7ae-tls-certs\") pod \"e2e-same-id-b-d3c310b5-kserve-5c98bfdf87-6sm2w\" (UID: \"bb4ed0f4-d684-42c4-9bf2-295f7250e7ae\") " pod="llm/e2e-same-id-b-d3c310b5-kserve-5c98bfdf87-6sm2w"
Sep 17 14:38:27.688606 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:38:27.688548    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb4ed0f4-d684-42c4-9bf2-295f7250e7ae-dshm\") pod \"e2e-same-id-b-d3c310b5-kserve-5c98bfdf87-6sm2w\" (UID: \"bb4ed0f4-d684-42c4-9bf2-295f7250e7ae\") " pod="llm/e2e-same-id-b-d3c310b5-kserve-5c98bfdf87-6sm2w"
Sep 17 14:38:27.688606 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:38:27.688588    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb4ed0f4-d684-42c4-9bf2-295f7250e7ae-model-cache\") pod \"e2e-same-id-b-d3c310b5-kserve-5c98bfdf87-6sm2w\" (UID: \"bb4ed0f4-d684-42c4-9bf2-295f7250e7ae\") " pod="llm/e2e-same-id-b-d3c310b5-kserve-5c98bfdf87-6sm2w"
Sep 17 14:38:27.688787 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:38:27.688650    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb4ed0f4-d684-42c4-9bf2-295f7250e7ae-tmp-dir\") pod \"e2e-same-id-b-d3c310b5-kserve-5c98bfdf87-6sm2w\" (UID: \"bb4ed0f4-d684-42c4-9bf2-295f7250e7ae\") " pod="llm/e2e-same-id-b-d3c310b5-kserve-5c98bfdf87-6sm2w"
Sep 17 14:38:27.688787 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:38:27.688680    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb4ed0f4-d684-42c4-9bf2-295f7250e7ae-home\") pod \"e2e-same-id-b-d3c310b5-kserve-5c98bfdf87-6sm2w\" (UID: \"bb4ed0f4-d684-42c4-9bf2-295f7250e7ae\") " pod="llm/e2e-same-id-b-d3c310b5-kserve-5c98bfdf87-6sm2w"
Sep 17 14:38:27.700848 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:38:27.700816    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:38:27.789505 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:38:27.789451    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlxkc\" (UniqueName: \"kubernetes.io/projected/bb4ed0f4-d684-42c4-9bf2-295f7250e7ae-kube-api-access-tlxkc\") pod \"e2e-same-id-b-d3c310b5-kserve-5c98bfdf87-6sm2w\" (UID: \"bb4ed0f4-d684-42c4-9bf2-295f7250e7ae\") " pod="llm/e2e-same-id-b-d3c310b5-kserve-5c98bfdf87-6sm2w"
Sep 17 14:38:27.789708 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:38:27.789596    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb4ed0f4-d684-42c4-9bf2-295f7250e7ae-tls-certs\") pod \"e2e-same-id-b-d3c310b5-kserve-5c98bfdf87-6sm2w\" (UID: \"bb4ed0f4-d684-42c4-9bf2-295f7250e7ae\") " pod="llm/e2e-same-id-b-d3c310b5-kserve-5c98bfdf87-6sm2w"
Sep 17 14:38:27.789708 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:38:27.789628    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb4ed0f4-d684-42c4-9bf2-295f7250e7ae-dshm\") pod \"e2e-same-id-b-d3c310b5-kserve-5c98bfdf87-6sm2w\" (UID: \"bb4ed0f4-d684-42c4-9bf2-295f7250e7ae\") " pod="llm/e2e-same-id-b-d3c310b5-kserve-5c98bfdf87-6sm2w"
Sep 17 14:38:27.789708 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:38:27.789650    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb4ed0f4-d684-42c4-9bf2-295f7250e7ae-model-cache\") pod \"e2e-same-id-b-d3c310b5-kserve-5c98bfdf87-6sm2w\" (UID: \"bb4ed0f4-d684-42c4-9bf2-295f7250e7ae\") " pod="llm/e2e-same-id-b-d3c310b5-kserve-5c98bfdf87-6sm2w"
Sep 17 14:38:27.789708 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:38:27.789682    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb4ed0f4-d684-42c4-9bf2-295f7250e7ae-tmp-dir\") pod \"e2e-same-id-b-d3c310b5-kserve-5c98bfdf87-6sm2w\" (UID: \"bb4ed0f4-d684-42c4-9bf2-295f7250e7ae\") " pod="llm/e2e-same-id-b-d3c310b5-kserve-5c98bfdf87-6sm2w"
Sep 17 14:38:27.789929 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:38:27.789810    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb4ed0f4-d684-42c4-9bf2-295f7250e7ae-home\") pod \"e2e-same-id-b-d3c310b5-kserve-5c98bfdf87-6sm2w\" (UID: \"bb4ed0f4-d684-42c4-9bf2-295f7250e7ae\") " pod="llm/e2e-same-id-b-d3c310b5-kserve-5c98bfdf87-6sm2w"
Sep 17 14:38:27.790155 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:38:27.790134    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb4ed0f4-d684-42c4-9bf2-295f7250e7ae-model-cache\") pod \"e2e-same-id-b-d3c310b5-kserve-5c98bfdf87-6sm2w\" (UID: \"bb4ed0f4-d684-42c4-9bf2-295f7250e7ae\") " pod="llm/e2e-same-id-b-d3c310b5-kserve-5c98bfdf87-6sm2w"
Sep 17 14:38:27.790223 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:38:27.790167    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb4ed0f4-d684-42c4-9bf2-295f7250e7ae-tmp-dir\") pod \"e2e-same-id-b-d3c310b5-kserve-5c98bfdf87-6sm2w\" (UID: \"bb4ed0f4-d684-42c4-9bf2-295f7250e7ae\") " pod="llm/e2e-same-id-b-d3c310b5-kserve-5c98bfdf87-6sm2w"
Sep 17 14:38:27.790223 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:38:27.790207    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb4ed0f4-d684-42c4-9bf2-295f7250e7ae-home\") pod \"e2e-same-id-b-d3c310b5-kserve-5c98bfdf87-6sm2w\" (UID: \"bb4ed0f4-d684-42c4-9bf2-295f7250e7ae\") " pod="llm/e2e-same-id-b-d3c310b5-kserve-5c98bfdf87-6sm2w"
Sep 17 14:38:27.791958 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:38:27.791930    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb4ed0f4-d684-42c4-9bf2-295f7250e7ae-dshm\") pod \"e2e-same-id-b-d3c310b5-kserve-5c98bfdf87-6sm2w\" (UID: \"bb4ed0f4-d684-42c4-9bf2-295f7250e7ae\") " pod="llm/e2e-same-id-b-d3c310b5-kserve-5c98bfdf87-6sm2w"
Sep 17 14:38:27.792197 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:38:27.792180    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb4ed0f4-d684-42c4-9bf2-295f7250e7ae-tls-certs\") pod \"e2e-same-id-b-d3c310b5-kserve-5c98bfdf87-6sm2w\" (UID: \"bb4ed0f4-d684-42c4-9bf2-295f7250e7ae\") " pod="llm/e2e-same-id-b-d3c310b5-kserve-5c98bfdf87-6sm2w"
Sep 17 14:38:27.827836 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:38:27.827808    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlxkc\" (UniqueName: \"kubernetes.io/projected/bb4ed0f4-d684-42c4-9bf2-295f7250e7ae-kube-api-access-tlxkc\") pod \"e2e-same-id-b-d3c310b5-kserve-5c98bfdf87-6sm2w\" (UID: \"bb4ed0f4-d684-42c4-9bf2-295f7250e7ae\") " pod="llm/e2e-same-id-b-d3c310b5-kserve-5c98bfdf87-6sm2w"
Sep 17 14:38:27.910992 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:38:27.910903    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-d3c310b5-kserve-5c98bfdf87-6sm2w"
Sep 17 14:38:28.188488 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:38:28.188433    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-d3c310b5-kserve-5c98bfdf87-6sm2w"]
Sep 17 14:38:28.190347 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:38:28.190320    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb4ed0f4_d684_42c4_9bf2_295f7250e7ae.slice/crio-73896f7efb13a57fd74ce9d013f29944b097181b2d9840c48abe634dda841a73 WatchSource:0}: Error finding container 73896f7efb13a57fd74ce9d013f29944b097181b2d9840c48abe634dda841a73: Status 404 returned error can't find the container with id 73896f7efb13a57fd74ce9d013f29944b097181b2d9840c48abe634dda841a73
Sep 17 14:38:28.224070 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:38:28.224043    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-d3c310b5-kserve-5c98bfdf87-6sm2w" event={"ID":"bb4ed0f4-d684-42c4-9bf2-295f7250e7ae","Type":"ContainerStarted","Data":"73896f7efb13a57fd74ce9d013f29944b097181b2d9840c48abe634dda841a73"}
Sep 17 14:38:29.231877 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:38:29.231836    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-d3c310b5-kserve-5c98bfdf87-6sm2w" event={"ID":"bb4ed0f4-d684-42c4-9bf2-295f7250e7ae","Type":"ContainerStarted","Data":"1127ca0d6b3722d00de686e5ba49a72b44579ef90ba317e5b4f885f8fbc05f4a"}
Sep 17 14:38:29.258189 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:38:29.258134    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-d3c310b5-kserve-5c98bfdf87-6sm2w" podStartSLOduration=2.258117673 podStartE2EDuration="2.258117673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:38:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:38:29.256551426 +0000 UTC m=+2578.198695071" watchObservedRunningTime="2026-09-17 14:38:29.258117673 +0000 UTC m=+2578.200261344"
Sep 17 14:38:32.369919 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:38:32.369889    2560 scope.go:117] "RemoveContainer" containerID="087303a4071a9a6f630ba6adeadfd6c886a650fdc64d7a982e61ecb3ac6f31d0"
Sep 17 14:38:32.378574 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:38:32.378553    2560 scope.go:117] "RemoveContainer" containerID="9861848a84c6d7a4a774fb46e165129821225efde7ad9e9bf543f200222abdda"
Sep 17 14:38:37.911653 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:38:37.911619    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-d3c310b5-kserve-5c98bfdf87-6sm2w"
Sep 17 14:38:37.911653 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:38:37.911659    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-d3c310b5-kserve-5c98bfdf87-6sm2w"
Sep 17 14:38:37.923837 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:38:37.923810    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-d3c310b5-kserve-5c98bfdf87-6sm2w"
Sep 17 14:38:38.292716 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:38:38.292640    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-d3c310b5-kserve-5c98bfdf87-6sm2w"
Sep 17 14:38:47.809492 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:38:47.809446    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:38:49.872392 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:38:49.872360    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-d3c310b5-kserve-5c98bfdf87-6sm2w"]
Sep 17 14:38:49.896753 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:38:49.896727    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:39:04.891229 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:04.891166    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-d3c310b5-kserve-5c98bfdf87-6sm2w" podUID="bb4ed0f4-d684-42c4-9bf2-295f7250e7ae" containerName="main" containerID="cri-o://1127ca0d6b3722d00de686e5ba49a72b44579ef90ba317e5b4f885f8fbc05f4a" gracePeriod=45
Sep 17 14:39:05.138252 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:05.138224    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-d3c310b5-kserve-5c98bfdf87-6sm2w"
Sep 17 14:39:05.215989 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:05.215926    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb4ed0f4-d684-42c4-9bf2-295f7250e7ae-tls-certs\") pod \"bb4ed0f4-d684-42c4-9bf2-295f7250e7ae\" (UID: \"bb4ed0f4-d684-42c4-9bf2-295f7250e7ae\") "
Sep 17 14:39:05.215989 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:05.215964    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb4ed0f4-d684-42c4-9bf2-295f7250e7ae-home\") pod \"bb4ed0f4-d684-42c4-9bf2-295f7250e7ae\" (UID: \"bb4ed0f4-d684-42c4-9bf2-295f7250e7ae\") "
Sep 17 14:39:05.216157 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:05.216020    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tlxkc\" (UniqueName: \"kubernetes.io/projected/bb4ed0f4-d684-42c4-9bf2-295f7250e7ae-kube-api-access-tlxkc\") pod \"bb4ed0f4-d684-42c4-9bf2-295f7250e7ae\" (UID: \"bb4ed0f4-d684-42c4-9bf2-295f7250e7ae\") "
Sep 17 14:39:05.216157 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:05.216048    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb4ed0f4-d684-42c4-9bf2-295f7250e7ae-model-cache\") pod \"bb4ed0f4-d684-42c4-9bf2-295f7250e7ae\" (UID: \"bb4ed0f4-d684-42c4-9bf2-295f7250e7ae\") "
Sep 17 14:39:05.216157 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:05.216067    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb4ed0f4-d684-42c4-9bf2-295f7250e7ae-dshm\") pod \"bb4ed0f4-d684-42c4-9bf2-295f7250e7ae\" (UID: \"bb4ed0f4-d684-42c4-9bf2-295f7250e7ae\") "
Sep 17 14:39:05.216157 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:05.216132    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb4ed0f4-d684-42c4-9bf2-295f7250e7ae-tmp-dir\") pod \"bb4ed0f4-d684-42c4-9bf2-295f7250e7ae\" (UID: \"bb4ed0f4-d684-42c4-9bf2-295f7250e7ae\") "
Sep 17 14:39:05.216365 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:05.216308    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb4ed0f4-d684-42c4-9bf2-295f7250e7ae-home" (OuterVolumeSpecName: "home") pod "bb4ed0f4-d684-42c4-9bf2-295f7250e7ae" (UID: "bb4ed0f4-d684-42c4-9bf2-295f7250e7ae"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:39:05.216365 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:05.216320    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb4ed0f4-d684-42c4-9bf2-295f7250e7ae-model-cache" (OuterVolumeSpecName: "model-cache") pod "bb4ed0f4-d684-42c4-9bf2-295f7250e7ae" (UID: "bb4ed0f4-d684-42c4-9bf2-295f7250e7ae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:39:05.216534 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:05.216503    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb4ed0f4-d684-42c4-9bf2-295f7250e7ae-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bb4ed0f4-d684-42c4-9bf2-295f7250e7ae" (UID: "bb4ed0f4-d684-42c4-9bf2-295f7250e7ae"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:39:05.216609 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:05.216520    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb4ed0f4-d684-42c4-9bf2-295f7250e7ae-model-cache\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:39:05.216609 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:05.216562    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb4ed0f4-d684-42c4-9bf2-295f7250e7ae-home\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:39:05.218181 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:05.218159    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb4ed0f4-d684-42c4-9bf2-295f7250e7ae-kube-api-access-tlxkc" (OuterVolumeSpecName: "kube-api-access-tlxkc") pod "bb4ed0f4-d684-42c4-9bf2-295f7250e7ae" (UID: "bb4ed0f4-d684-42c4-9bf2-295f7250e7ae"). InnerVolumeSpecName "kube-api-access-tlxkc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:39:05.218260 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:05.218189    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb4ed0f4-d684-42c4-9bf2-295f7250e7ae-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bb4ed0f4-d684-42c4-9bf2-295f7250e7ae" (UID: "bb4ed0f4-d684-42c4-9bf2-295f7250e7ae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:39:05.218260 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:05.218226    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb4ed0f4-d684-42c4-9bf2-295f7250e7ae-dshm" (OuterVolumeSpecName: "dshm") pod "bb4ed0f4-d684-42c4-9bf2-295f7250e7ae" (UID: "bb4ed0f4-d684-42c4-9bf2-295f7250e7ae"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:39:05.317898 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:05.317865    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlxkc\" (UniqueName: \"kubernetes.io/projected/bb4ed0f4-d684-42c4-9bf2-295f7250e7ae-kube-api-access-tlxkc\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:39:05.317898 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:05.317895    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb4ed0f4-d684-42c4-9bf2-295f7250e7ae-dshm\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:39:05.317898 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:05.317904    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb4ed0f4-d684-42c4-9bf2-295f7250e7ae-tmp-dir\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:39:05.318067 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:05.317913    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb4ed0f4-d684-42c4-9bf2-295f7250e7ae-tls-certs\") on node \"ip-10-0-129-158.ec2.internal\" DevicePath \"\""
Sep 17 14:39:05.435866 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:05.435833    2560 generic.go:356] "Generic (PLEG): container finished" podID="bb4ed0f4-d684-42c4-9bf2-295f7250e7ae" containerID="1127ca0d6b3722d00de686e5ba49a72b44579ef90ba317e5b4f885f8fbc05f4a" exitCode=0
Sep 17 14:39:05.435997 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:05.435881    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-d3c310b5-kserve-5c98bfdf87-6sm2w"
Sep 17 14:39:05.435997 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:05.435926    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-d3c310b5-kserve-5c98bfdf87-6sm2w" event={"ID":"bb4ed0f4-d684-42c4-9bf2-295f7250e7ae","Type":"ContainerDied","Data":"1127ca0d6b3722d00de686e5ba49a72b44579ef90ba317e5b4f885f8fbc05f4a"}
Sep 17 14:39:05.435997 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:05.435964    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-d3c310b5-kserve-5c98bfdf87-6sm2w" event={"ID":"bb4ed0f4-d684-42c4-9bf2-295f7250e7ae","Type":"ContainerDied","Data":"73896f7efb13a57fd74ce9d013f29944b097181b2d9840c48abe634dda841a73"}
Sep 17 14:39:05.435997 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:05.435982    2560 scope.go:117] "RemoveContainer" containerID="1127ca0d6b3722d00de686e5ba49a72b44579ef90ba317e5b4f885f8fbc05f4a"
Sep 17 14:39:05.445777 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:05.445761    2560 scope.go:117] "RemoveContainer" containerID="1127ca0d6b3722d00de686e5ba49a72b44579ef90ba317e5b4f885f8fbc05f4a"
Sep 17 14:39:05.446042 ip-10-0-129-158 kubenswrapper[2560]: E0917 14:39:05.446025    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1127ca0d6b3722d00de686e5ba49a72b44579ef90ba317e5b4f885f8fbc05f4a\": container with ID starting with 1127ca0d6b3722d00de686e5ba49a72b44579ef90ba317e5b4f885f8fbc05f4a not found: ID does not exist" containerID="1127ca0d6b3722d00de686e5ba49a72b44579ef90ba317e5b4f885f8fbc05f4a"
Sep 17 14:39:05.446106 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:05.446050    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1127ca0d6b3722d00de686e5ba49a72b44579ef90ba317e5b4f885f8fbc05f4a"} err="failed to get container status \"1127ca0d6b3722d00de686e5ba49a72b44579ef90ba317e5b4f885f8fbc05f4a\": rpc error: code = NotFound desc = could not find container \"1127ca0d6b3722d00de686e5ba49a72b44579ef90ba317e5b4f885f8fbc05f4a\": container with ID starting with 1127ca0d6b3722d00de686e5ba49a72b44579ef90ba317e5b4f885f8fbc05f4a not found: ID does not exist"
Sep 17 14:39:05.457394 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:05.457371    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-d3c310b5-kserve-5c98bfdf87-6sm2w"]
Sep 17 14:39:05.462225 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:05.462207    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-d3c310b5-kserve-5c98bfdf87-6sm2w"]
Sep 17 14:39:05.571218 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:05.571189    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb4ed0f4-d684-42c4-9bf2-295f7250e7ae" path="/var/lib/kubelet/pods/bb4ed0f4-d684-42c4-9bf2-295f7250e7ae/volumes"
Sep 17 14:39:32.402608 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:32.402573    2560 scope.go:117] "RemoveContainer" containerID="f6c17f1e82d1682f13f3c2e0f1ae3f2ab118fcae8646197e6bcd8d43e734e0a3"
Sep 17 14:39:32.411223 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:32.411205    2560 scope.go:117] "RemoveContainer" containerID="2833fdbb1019d969c8c9338902cea329d6120e56bf01d4c12e7eba9b575873ff"
Sep 17 14:39:36.703163 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:36.703132    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-8ktvg_f7aa5922-4de4-489a-ac3f-bc61b40034bb/server/0.log"
Sep 17 14:39:36.827473 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:36.827445    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-n5wzq_396ab0f1-ec8d-4a85-891f-7730c439051a/manager/0.log"
Sep 17 14:39:37.233395 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:37.233360    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-fcc687555-sjzts_ca6e8239-2e88-4499-b547-d2bc9be28568/copy-manifests/0.log"
Sep 17 14:39:37.240499 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:37.240479    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-fcc687555-sjzts_ca6e8239-2e88-4499-b547-d2bc9be28568/manager/0.log"
Sep 17 14:39:37.827666 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:37.827634    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29827575-cfsg7_3792946f-bb55-422c-9182-9be8a911a6cd/cleanup/0.log"
Sep 17 14:39:37.935570 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:37.935546    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29827590-p8rkr_4896d33f-f407-4db7-9473-75db5411ffb5/cleanup/0.log"
Sep 17 14:39:39.943558 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:39.943525    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cwmmzm_06ee2c26-2d04-4fa7-8e5d-a70ffe53d2b8/istio-proxy/0.log"
Sep 17 14:39:40.171925 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:40.171898    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-66d75c69df-wz6ps_7cdbd220-7e88-4dcf-976f-31b723414622/kube-auth-proxy/0.log"
Sep 17 14:39:40.775862 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:40.775831    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-997fdcdb-82rfx_644fc778-b087-473e-91bf-44f9a89c2fef/router/0.log"
Sep 17 14:39:41.120300 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:41.120275    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-gdg6k_1873d7b7-3c5b-46be-b6f1-757eb8c3b2e0/main/0.log"
Sep 17 14:39:41.234060 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:41.234037    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-msqdt_68fc6c1d-a850-4223-be13-27c9d4c47625/main/0.log"
Sep 17 14:39:41.727343 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:41.727317    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-fdz9h_9bf20d2c-b82c-4536-8f83-f64992c0bb32/main/0.log"
Sep 17 14:39:47.428881 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:47.428851    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-g9q75/must-gather-5j857"]
Sep 17 14:39:47.429357 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:47.429257    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb4ed0f4-d684-42c4-9bf2-295f7250e7ae" containerName="main"
Sep 17 14:39:47.429357 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:47.429270    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb4ed0f4-d684-42c4-9bf2-295f7250e7ae" containerName="main"
Sep 17 14:39:47.429357 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:47.429352    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="00fdb0ea-d9f7-4dba-9e77-f26c85e50e0c" containerName="istio-proxy"
Sep 17 14:39:47.429541 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:47.429375    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb4ed0f4-d684-42c4-9bf2-295f7250e7ae" containerName="main"
Sep 17 14:39:47.429541 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:47.429394    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:39:47.432549 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:47.432534    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g9q75/must-gather-5j857"
Sep 17 14:39:47.436343 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:47.436320    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-g9q75\"/\"kube-root-ca.crt\""
Sep 17 14:39:47.436823 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:47.436801    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-g9q75\"/\"default-dockercfg-9jrgw\""
Sep 17 14:39:47.436823 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:47.436801    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-g9q75\"/\"openshift-service-ca.crt\""
Sep 17 14:39:47.455973 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:47.455949    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g9q75/must-gather-5j857"]
Sep 17 14:39:47.500394 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:47.500371    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log" limit=1024
Sep 17 14:39:47.563033 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:47.563009    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/efff0998-2e86-4337-b272-f589f6c24fc4-must-gather-output\") pod \"must-gather-5j857\" (UID: \"efff0998-2e86-4337-b272-f589f6c24fc4\") " pod="openshift-must-gather-g9q75/must-gather-5j857"
Sep 17 14:39:47.563133 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:47.563050    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwd72\" (UniqueName: \"kubernetes.io/projected/efff0998-2e86-4337-b272-f589f6c24fc4-kube-api-access-hwd72\") pod \"must-gather-5j857\" (UID: \"efff0998-2e86-4337-b272-f589f6c24fc4\") " pod="openshift-must-gather-g9q75/must-gather-5j857"
Sep 17 14:39:47.663520 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:47.663498    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/efff0998-2e86-4337-b272-f589f6c24fc4-must-gather-output\") pod \"must-gather-5j857\" (UID: \"efff0998-2e86-4337-b272-f589f6c24fc4\") " pod="openshift-must-gather-g9q75/must-gather-5j857"
Sep 17 14:39:47.663648 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:47.663532    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwd72\" (UniqueName: \"kubernetes.io/projected/efff0998-2e86-4337-b272-f589f6c24fc4-kube-api-access-hwd72\") pod \"must-gather-5j857\" (UID: \"efff0998-2e86-4337-b272-f589f6c24fc4\") " pod="openshift-must-gather-g9q75/must-gather-5j857"
Sep 17 14:39:47.663850 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:47.663832    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/efff0998-2e86-4337-b272-f589f6c24fc4-must-gather-output\") pod \"must-gather-5j857\" (UID: \"efff0998-2e86-4337-b272-f589f6c24fc4\") " pod="openshift-must-gather-g9q75/must-gather-5j857"
Sep 17 14:39:47.672221 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:47.672202    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwd72\" (UniqueName: \"kubernetes.io/projected/efff0998-2e86-4337-b272-f589f6c24fc4-kube-api-access-hwd72\") pod \"must-gather-5j857\" (UID: \"efff0998-2e86-4337-b272-f589f6c24fc4\") " pod="openshift-must-gather-g9q75/must-gather-5j857"
Sep 17 14:39:47.741722 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:47.741662    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g9q75/must-gather-5j857"
Sep 17 14:39:48.027178 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:48.027152    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g9q75/must-gather-5j857"]
Sep 17 14:39:48.029316 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:39:48.029290    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podefff0998_2e86_4337_b272_f589f6c24fc4.slice/crio-275d84cd8b573d10eeff59e6cd5e468e7fc7b941ac39cbfe54dfc983862b7c54 WatchSource:0}: Error finding container 275d84cd8b573d10eeff59e6cd5e468e7fc7b941ac39cbfe54dfc983862b7c54: Status 404 returned error can't find the container with id 275d84cd8b573d10eeff59e6cd5e468e7fc7b941ac39cbfe54dfc983862b7c54
Sep 17 14:39:48.031249 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:48.031230    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 14:39:48.681516 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:48.681486    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g9q75/must-gather-5j857" event={"ID":"efff0998-2e86-4337-b272-f589f6c24fc4","Type":"ContainerStarted","Data":"275d84cd8b573d10eeff59e6cd5e468e7fc7b941ac39cbfe54dfc983862b7c54"}
Sep 17 14:39:49.693488 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:49.693431    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g9q75/must-gather-5j857" event={"ID":"efff0998-2e86-4337-b272-f589f6c24fc4","Type":"ContainerStarted","Data":"9343a0bfe68593a22a0b2abd3b17721f3b0d52d5ca36e893788471c9e24f802d"}
Sep 17 14:39:49.693953 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:49.693509    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g9q75/must-gather-5j857" event={"ID":"efff0998-2e86-4337-b272-f589f6c24fc4","Type":"ContainerStarted","Data":"60066d9bbd6d70c83403345779283c5addb1661b0c4a86c736a9ca948125a7fd"}
Sep 17 14:39:49.721192 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:49.721130    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-g9q75/must-gather-5j857" podStartSLOduration=2.034609573 podStartE2EDuration="2.721109114s" totalImagesPullingTime="686.499541ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:39:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:39:49.715598819 +0000 UTC m=+2658.657742465" watchObservedRunningTime="2026-09-17 14:39:49.721109114 +0000 UTC m=+2658.663252761"
Sep 17 14:39:50.414519 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:50.414485    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-58g8x_14057e5c-c9fb-4894-869c-9780abc9c769/global-pull-secret-syncer/0.log"
Sep 17 14:39:50.568272 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:50.568235    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-gqsqk_eae24b6c-3322-4165-96fa-cd61dab923ca/konnectivity-agent/0.log"
Sep 17 14:39:50.699897 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:50.699821    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-158.ec2.internal_9a84bb5c5cbac6dbd226621c20c98385/haproxy/0.log"
Sep 17 14:39:57.625033 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:57.624991    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0327e6ab-d63a-438a-b925-fc60d270bc4d/alertmanager/0.log"
Sep 17 14:39:57.658564 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:57.658535    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0327e6ab-d63a-438a-b925-fc60d270bc4d/config-reloader/0.log"
Sep 17 14:39:57.690936 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:57.690881    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0327e6ab-d63a-438a-b925-fc60d270bc4d/kube-rbac-proxy-web/0.log"
Sep 17 14:39:57.728441 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:57.728407    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0327e6ab-d63a-438a-b925-fc60d270bc4d/kube-rbac-proxy/0.log"
Sep 17 14:39:57.759637 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:57.759542    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0327e6ab-d63a-438a-b925-fc60d270bc4d/kube-rbac-proxy-metric/0.log"
Sep 17 14:39:57.793115 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:57.793084    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0327e6ab-d63a-438a-b925-fc60d270bc4d/prom-label-proxy/0.log"
Sep 17 14:39:57.829247 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:57.829215    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0327e6ab-d63a-438a-b925-fc60d270bc4d/init-config-reloader/0.log"
Sep 17 14:39:57.923766 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:57.923738    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-sdzrz_a35f644c-4064-4099-8721-d796bddb0644/kube-state-metrics/0.log"
Sep 17 14:39:57.974397 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:57.974348    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-sdzrz_a35f644c-4064-4099-8721-d796bddb0644/kube-rbac-proxy-main/0.log"
Sep 17 14:39:58.040354 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:58.040269    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-sdzrz_a35f644c-4064-4099-8721-d796bddb0644/kube-rbac-proxy-self/0.log"
Sep 17 14:39:58.166491 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:58.166444    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-gdznh_6b5113b4-136d-4d6c-a435-b0d9e7ea6232/monitoring-plugin/0.log"
Sep 17 14:39:58.317735 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:58.317708    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gqkh8_7ae12eb6-f825-4313-8884-1e879544e4f6/node-exporter/0.log"
Sep 17 14:39:58.345117 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:58.345085    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gqkh8_7ae12eb6-f825-4313-8884-1e879544e4f6/kube-rbac-proxy/0.log"
Sep 17 14:39:58.367369 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:58.367342    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gqkh8_7ae12eb6-f825-4313-8884-1e879544e4f6/init-textfile/0.log"
Sep 17 14:39:58.489984 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:58.489952    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-blrzn_b89d8a9d-a0d5-4578-bf8a-1711c9e5646a/kube-rbac-proxy-main/0.log"
Sep 17 14:39:58.525360 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:58.525326    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-blrzn_b89d8a9d-a0d5-4578-bf8a-1711c9e5646a/kube-rbac-proxy-self/0.log"
Sep 17 14:39:58.549178 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:58.549147    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-blrzn_b89d8a9d-a0d5-4578-bf8a-1711c9e5646a/openshift-state-metrics/0.log"
Sep 17 14:39:58.789448 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:58.789354    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-wdgwn_f43dc725-01fa-4d2c-8762-5053835093a3/prometheus-operator/0.log"
Sep 17 14:39:58.810929 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:58.810890    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-wdgwn_f43dc725-01fa-4d2c-8762-5053835093a3/kube-rbac-proxy/0.log"
Sep 17 14:39:58.841438 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:58.841391    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-d5l5x_55763440-e26d-49dd-b74f-da96ae2d99d9/prometheus-operator-admission-webhook/0.log"
Sep 17 14:39:58.875500 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:58.875446    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-56cd85c7b4-4cj6v_90da0389-baf3-4223-8975-ff83b7d02a71/telemeter-client/0.log"
Sep 17 14:39:58.903939 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:58.903908    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-56cd85c7b4-4cj6v_90da0389-baf3-4223-8975-ff83b7d02a71/reload/0.log"
Sep 17 14:39:58.933793 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:58.933760    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-56cd85c7b4-4cj6v_90da0389-baf3-4223-8975-ff83b7d02a71/kube-rbac-proxy/0.log"
Sep 17 14:39:58.979659 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:58.979599    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-754b4f6bdc-wngxz_6ee51de3-c49d-45aa-addb-41d96f782d1e/thanos-query/0.log"
Sep 17 14:39:59.009406 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:59.009360    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-754b4f6bdc-wngxz_6ee51de3-c49d-45aa-addb-41d96f782d1e/kube-rbac-proxy-web/0.log"
Sep 17 14:39:59.034777 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:59.034752    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-754b4f6bdc-wngxz_6ee51de3-c49d-45aa-addb-41d96f782d1e/kube-rbac-proxy/0.log"
Sep 17 14:39:59.057999 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:59.057977    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-754b4f6bdc-wngxz_6ee51de3-c49d-45aa-addb-41d96f782d1e/prom-label-proxy/0.log"
Sep 17 14:39:59.083677 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:59.083650    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-754b4f6bdc-wngxz_6ee51de3-c49d-45aa-addb-41d96f782d1e/kube-rbac-proxy-rules/0.log"
Sep 17 14:39:59.112628 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:59.112599    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-754b4f6bdc-wngxz_6ee51de3-c49d-45aa-addb-41d96f782d1e/kube-rbac-proxy-metrics/0.log"
Sep 17 14:39:59.577076 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:59.577044    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-g9q75/perf-node-gather-daemonset-k4dck"]
Sep 17 14:39:59.582742 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:59.582712    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g9q75/perf-node-gather-daemonset-k4dck"
Sep 17 14:39:59.593869 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:59.593795    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g9q75/perf-node-gather-daemonset-k4dck"]
Sep 17 14:39:59.597569 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:59.595760    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02e0ec9f-0eed-4210-ab77-63f0059bfc84-sys\") pod \"perf-node-gather-daemonset-k4dck\" (UID: \"02e0ec9f-0eed-4210-ab77-63f0059bfc84\") " pod="openshift-must-gather-g9q75/perf-node-gather-daemonset-k4dck"
Sep 17 14:39:59.597569 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:59.595902    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhjkb\" (UniqueName: \"kubernetes.io/projected/02e0ec9f-0eed-4210-ab77-63f0059bfc84-kube-api-access-fhjkb\") pod \"perf-node-gather-daemonset-k4dck\" (UID: \"02e0ec9f-0eed-4210-ab77-63f0059bfc84\") " pod="openshift-must-gather-g9q75/perf-node-gather-daemonset-k4dck"
Sep 17 14:39:59.597569 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:59.595937    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/02e0ec9f-0eed-4210-ab77-63f0059bfc84-lib-modules\") pod \"perf-node-gather-daemonset-k4dck\" (UID: \"02e0ec9f-0eed-4210-ab77-63f0059bfc84\") " pod="openshift-must-gather-g9q75/perf-node-gather-daemonset-k4dck"
Sep 17 14:39:59.597569 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:59.595962    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/02e0ec9f-0eed-4210-ab77-63f0059bfc84-podres\") pod \"perf-node-gather-daemonset-k4dck\" (UID: \"02e0ec9f-0eed-4210-ab77-63f0059bfc84\") " pod="openshift-must-gather-g9q75/perf-node-gather-daemonset-k4dck"
Sep 17 14:39:59.597569 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:59.596068    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/02e0ec9f-0eed-4210-ab77-63f0059bfc84-proc\") pod \"perf-node-gather-daemonset-k4dck\" (UID: \"02e0ec9f-0eed-4210-ab77-63f0059bfc84\") " pod="openshift-must-gather-g9q75/perf-node-gather-daemonset-k4dck"
Sep 17 14:39:59.697348 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:59.697285    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhjkb\" (UniqueName: \"kubernetes.io/projected/02e0ec9f-0eed-4210-ab77-63f0059bfc84-kube-api-access-fhjkb\") pod \"perf-node-gather-daemonset-k4dck\" (UID: \"02e0ec9f-0eed-4210-ab77-63f0059bfc84\") " pod="openshift-must-gather-g9q75/perf-node-gather-daemonset-k4dck"
Sep 17 14:39:59.697683 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:59.697660    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/02e0ec9f-0eed-4210-ab77-63f0059bfc84-lib-modules\") pod \"perf-node-gather-daemonset-k4dck\" (UID: \"02e0ec9f-0eed-4210-ab77-63f0059bfc84\") " pod="openshift-must-gather-g9q75/perf-node-gather-daemonset-k4dck"
Sep 17 14:39:59.697820 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:59.697803    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/02e0ec9f-0eed-4210-ab77-63f0059bfc84-podres\") pod \"perf-node-gather-daemonset-k4dck\" (UID: \"02e0ec9f-0eed-4210-ab77-63f0059bfc84\") " pod="openshift-must-gather-g9q75/perf-node-gather-daemonset-k4dck"
Sep 17 14:39:59.697987 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:59.697961    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/02e0ec9f-0eed-4210-ab77-63f0059bfc84-podres\") pod \"perf-node-gather-daemonset-k4dck\" (UID: \"02e0ec9f-0eed-4210-ab77-63f0059bfc84\") " pod="openshift-must-gather-g9q75/perf-node-gather-daemonset-k4dck"
Sep 17 14:39:59.698083 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:59.697844    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/02e0ec9f-0eed-4210-ab77-63f0059bfc84-lib-modules\") pod \"perf-node-gather-daemonset-k4dck\" (UID: \"02e0ec9f-0eed-4210-ab77-63f0059bfc84\") " pod="openshift-must-gather-g9q75/perf-node-gather-daemonset-k4dck"
Sep 17 14:39:59.698182 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:59.698168    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/02e0ec9f-0eed-4210-ab77-63f0059bfc84-proc\") pod \"perf-node-gather-daemonset-k4dck\" (UID: \"02e0ec9f-0eed-4210-ab77-63f0059bfc84\") " pod="openshift-must-gather-g9q75/perf-node-gather-daemonset-k4dck"
Sep 17 14:39:59.698298 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:59.698282    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02e0ec9f-0eed-4210-ab77-63f0059bfc84-sys\") pod \"perf-node-gather-daemonset-k4dck\" (UID: \"02e0ec9f-0eed-4210-ab77-63f0059bfc84\") " pod="openshift-must-gather-g9q75/perf-node-gather-daemonset-k4dck"
Sep 17 14:39:59.698536 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:59.698516    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02e0ec9f-0eed-4210-ab77-63f0059bfc84-sys\") pod \"perf-node-gather-daemonset-k4dck\" (UID: \"02e0ec9f-0eed-4210-ab77-63f0059bfc84\") " pod="openshift-must-gather-g9q75/perf-node-gather-daemonset-k4dck"
Sep 17 14:39:59.698733 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:59.698714    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/02e0ec9f-0eed-4210-ab77-63f0059bfc84-proc\") pod \"perf-node-gather-daemonset-k4dck\" (UID: \"02e0ec9f-0eed-4210-ab77-63f0059bfc84\") " pod="openshift-must-gather-g9q75/perf-node-gather-daemonset-k4dck"
Sep 17 14:39:59.719820 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:59.719786    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhjkb\" (UniqueName: \"kubernetes.io/projected/02e0ec9f-0eed-4210-ab77-63f0059bfc84-kube-api-access-fhjkb\") pod \"perf-node-gather-daemonset-k4dck\" (UID: \"02e0ec9f-0eed-4210-ab77-63f0059bfc84\") " pod="openshift-must-gather-g9q75/perf-node-gather-daemonset-k4dck"
Sep 17 14:39:59.902561 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:39:59.902440    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g9q75/perf-node-gather-daemonset-k4dck"
Sep 17 14:40:00.199355 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:40:00.199326    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g9q75/perf-node-gather-daemonset-k4dck"]
Sep 17 14:40:00.200981 ip-10-0-129-158 kubenswrapper[2560]: W0917 14:40:00.200936    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod02e0ec9f_0eed_4210_ab77_63f0059bfc84.slice/crio-1e6afa8144fa329c027a7c390890599f33608b9ea0d95209e2fd68c3af33e908 WatchSource:0}: Error finding container 1e6afa8144fa329c027a7c390890599f33608b9ea0d95209e2fd68c3af33e908: Status 404 returned error can't find the container with id 1e6afa8144fa329c027a7c390890599f33608b9ea0d95209e2fd68c3af33e908
Sep 17 14:40:00.786987 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:40:00.786948    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g9q75/perf-node-gather-daemonset-k4dck" event={"ID":"02e0ec9f-0eed-4210-ab77-63f0059bfc84","Type":"ContainerStarted","Data":"79caeba6a72cffd816a650a2f1fe39bb26a431dae748dfffc4db1a263a303d36"}
Sep 17 14:40:00.787193 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:40:00.787011    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g9q75/perf-node-gather-daemonset-k4dck" event={"ID":"02e0ec9f-0eed-4210-ab77-63f0059bfc84","Type":"ContainerStarted","Data":"1e6afa8144fa329c027a7c390890599f33608b9ea0d95209e2fd68c3af33e908"}
Sep 17 14:40:00.787193 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:40:00.787131    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-g9q75/perf-node-gather-daemonset-k4dck"
Sep 17 14:40:00.821397 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:40:00.821322    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-g9q75/perf-node-gather-daemonset-k4dck" podStartSLOduration=1.821301375 podStartE2EDuration="1.821301375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 14:39:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:40:00.819035718 +0000 UTC m=+2669.761179366" watchObservedRunningTime="2026-09-17 14:40:00.821301375 +0000 UTC m=+2669.763445025"
Sep 17 14:40:01.688390 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:40:01.688357    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5bbb6b5f44-9pjsb_3514d6f7-c94a-4c0a-9dfc-e15cad52cf68/console/0.log"
Sep 17 14:40:01.740204 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:40:01.740147    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-kpx6x_a2d73565-b83f-4582-9c80-84c460828fe6/download-server/0.log"
Sep 17 14:40:03.726563 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:40:03.726533    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-82dhq_45035743-85eb-4642-a5c9-adea5c5db4e1/dns/0.log"
Sep 17 14:40:03.750600 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:40:03.750576    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-82dhq_45035743-85eb-4642-a5c9-adea5c5db4e1/kube-rbac-proxy/0.log"
Sep 17 14:40:03.849028 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:40:03.848992    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2zqvg_a4a8010a-d12b-4b1a-a0db-e1c64109b151/dns-node-resolver/0.log"
Sep 17 14:40:04.621804 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:40:04.621767    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9wpxw_9fcb9893-714e-47b5-9b6f-37e706357ffe/node-ca/0.log"
Sep 17 14:40:05.799797 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:40:05.799764    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cwmmzm_06ee2c26-2d04-4fa7-8e5d-a70ffe53d2b8/istio-proxy/0.log"
Sep 17 14:40:06.653403 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:40:06.653355    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-66d75c69df-wz6ps_7cdbd220-7e88-4dcf-976f-31b723414622/kube-auth-proxy/0.log"
Sep 17 14:40:06.807809 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:40:06.807783    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-g9q75/perf-node-gather-daemonset-k4dck"
Sep 17 14:40:06.831086 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:40:06.831057    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-997fdcdb-82rfx_644fc778-b087-473e-91bf-44f9a89c2fef/router/0.log"
Sep 17 14:40:07.779896 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:40:07.779868    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wwqnd_2769cbbe-3a80-48c0-bf6b-a993360d1ca7/serve-healthcheck-canary/0.log"
Sep 17 14:40:08.635454 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:40:08.635425    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4848q_3e8dc684-ddfc-4801-a7de-7874e92eb31f/kube-rbac-proxy/0.log"
Sep 17 14:40:08.682192 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:40:08.682156    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4848q_3e8dc684-ddfc-4801-a7de-7874e92eb31f/exporter/0.log"
Sep 17 14:40:08.739407 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:40:08.739377    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4848q_3e8dc684-ddfc-4801-a7de-7874e92eb31f/extractor/0.log"
Sep 17 14:40:10.453111 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:40:10.453077    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cwmmzm_06ee2c26-2d04-4fa7-8e5d-a70ffe53d2b8/istio-proxy/0.log"
Sep 17 14:40:11.361220 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:40:11.361192    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-66d75c69df-wz6ps_7cdbd220-7e88-4dcf-976f-31b723414622/kube-auth-proxy/0.log"
Sep 17 14:40:11.504972 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:40:11.504941    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-997fdcdb-82rfx_644fc778-b087-473e-91bf-44f9a89c2fef/router/0.log"
Sep 17 14:40:12.133330 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:40:12.133297    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-8ktvg_f7aa5922-4de4-489a-ac3f-bc61b40034bb/server/0.log"
Sep 17 14:40:12.265416 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:40:12.265376    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-n5wzq_396ab0f1-ec8d-4a85-891f-7730c439051a/manager/0.log"
Sep 17 14:40:12.793169 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:40:12.793122    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-fcc687555-sjzts_ca6e8239-2e88-4499-b547-d2bc9be28568/manager/0.log"
Sep 17 14:40:12.849725 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:40:12.849696    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-fcc687555-sjzts_ca6e8239-2e88-4499-b547-d2bc9be28568/copy-manifests/0.log"
Sep 17 14:40:13.000660 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:40:13.000631    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cwmmzm_06ee2c26-2d04-4fa7-8e5d-a70ffe53d2b8/istio-proxy/0.log"
Sep 17 14:40:13.726236 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:40:13.726210    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-66d75c69df-wz6ps_7cdbd220-7e88-4dcf-976f-31b723414622/kube-auth-proxy/0.log"
Sep 17 14:40:14.010068 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:40:14.009995    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-997fdcdb-82rfx_644fc778-b087-473e-91bf-44f9a89c2fef/router/0.log"
Sep 17 14:40:14.377855 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:40:14.377829    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-69b649694d-jn5nd_c26ba51c-6096-49ae-9933-3c1f4398d5cf/manager/0.log"
Sep 17 14:40:14.467532 ip-10-0-129-158 kubenswrapper[2560]: I0917 14:40:14.467451    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-z4knd_531e646a-d3cf-4763-8547-6c48e8bdcc7b/openshift-lws-operator/0.log"